Installation

For teams 30 users or less, you can use Fluent for free, and can proceed with the installation instructions below after reading Fluent Free Trial License Agreement.

For teams larger than 30 users, you would need to contact us for a license key, and start a one-month free trial. If you already have a license key, you can proceed with the instructions below.

Fluent is a self-hosted, on-premise code navigator that lets you quickly and easily browse and search through large codebases. You configure it to mirror a list of repositories, and your Fluent server will pull them down every few minutes, index them, and make them available for fast searching and browsing.

If you need help or encounter problems at any step in this process, feel free to drop by the Fluent Chatroom or contact us at support@fluentcode.com.

Here are the steps in setting up Fluent:

1. Find a server to run Fluent on

Fluent is a self-hosted, on-premise application that runs on your own hardware. This could be within your company's own VPN.

The machine that's hosting Fluent will need free disk space equivalent to approximately twice the combined size of the repositories it's mirroring: to hold the repository mirrors, as well as the indices. Fluent also needs free memory about twice the combined size of the repositories it is hosting.

For example, a machine hosting 1gb worth of repositories will need about 2gb of free disk and 2gb of free memory.

2. Download the latest version of the Fluent server

3. Install Java 8 (update 101 or later) on the machine that will host Fluent

Java 8 is Fluent's only dependency. Instructions vary between platforms:

Make sure you are using update 101 (i.e. version 8u101) or later, as Fluent depends on Let's Encrypt SSL certificates which only become available starting that version.

4. Set up a Fluent's configuration file:

A minimal Fluent config.json looks like this:

{
    "repos": [
      {"url": "https://github.com/lihaoyi/Ammonite.git"}
    ],
    "port": 8080,
    "host": "localhost:880",
    "publicHost": ["localhost:8080"],
    "smtpHost": "aspmx.l.google.com",
    "validUsers": {
      "email": ["foo@gmail.com"]
    },
    "loginConfig": [
      {"$type": "fluent.web.EmailLogin"}
    ],
    "licenseKey": ""
}

Note that the config above supports on email login, though only for gmail accounts. It is configured to use Google's aspmx.l.google.com, which is a zero-setup free smtp server that can only send emails to @gmail.com accounts. If you are setting this up locally, or within a VPN, you can also use:

"loginConfig": [
  {"$type": "fluent.web.UnauthenticatedLogin"}
]

Which allows anyone who can access the website to login without any authentication. UnauthenticatedLogin is limited to 30 users, under Fluent's community edition. If you do use UnautheticatedLogin, it is your responsibility to make sure your server is within a VPN and not accessible to anyone who should not be able to see your code. For security reasons, we encourage you to use one of the other login mechanisms once you are done setting up your Fluent server.

If you wish to login using non-gmail email accounts, you would need to pick a different smtp host, e.g. smtp-relay.gmail.com (which needs to be configured at described here) or another third party host. If you wish to use Google or Github single sign-on, you would need to configure your Google or Github developer accounts to allow this as described below.

You need to substitute your own licenseKey in place of the placeholder shown above if you want to use Fluent for more than 30 users.

5. Run Fluent

You can use this config.json file to run fluent:

java -Xmx2g -jar fluent.jar config.json

This will run your Fluent server with 2GB of memory using the parameters specified in config.json. This runs Fluent interactively, with the logs streaming to STDOUT, and without HTTPS, but should be enough to let you test in your browser.

If you're done tweaking the configuration and want to leave your Fluent server for others to use, you will want to:

  • Keep Fluent running continuously across reboots and crashes via Upstart or similar

  • Set up Nginx or similar for SSL termination

Detailed Configuration

A more complex configuration may look like:

{
    "repos": [
      {"url": "https://github.com/lihaoyi/Ammonite.git"}
    ],
    "repoFolder": "target/repos",
    "indexCacheFolder": "target/indices",
    "port": 8080,
    "host": "localhost",
    "publicHost": ["localhost:8080"],
    "smtpHost": "aspmx.l.google.com",
    "validUsers": {
      "email": ["foo@bar.com", "qux@bar.com"],
      "github": ["baz", "githubuserid"]
    },
    "validUsersCommand": [],
    "loginConfig": [
      {"$type": "fluent.web.EmailLogin"},
      {
        "$type": "fluent.web.GoogleLogin",              
        "clientId": "123412341234-abcdefgabcdefgabcdefgabcdefgabcd.apps.googleusercontent.com",
        "clientSecret": "abcdef-abcedfghijklmnopq"
      },
      {
        "$type": "fluent.web.GithubLogin",
        "clientId": "abcedfabcdefabcdefab",
        "clientSecret": "abcdefabcdefabcdefabcdefabcdefabcdefabcd"
      }
    ],
    "licenseKey": "abcdef1234567890abcdef1234567890",
    "loginPrimaryMessages": [
      "A lightning-fast search engine for your codebase"
    ],
    "loginSecondaryMessages": [
      "Speed up your engineering team with Fluent"
    ]
}

The meaning of the keys is as follows:


repos

A list of repositories to mirror. The simplest case is when a repository is just a public URL to clone:

{"url": "https://github.com/lihaoyi/Ammonite.git"}

By default, Fluent will index and let you search the repository's master branch. You can specify a different indexed branch:

{
  "url": "https://github.com/MariaDB/server",
  "indexedBranches": [
    "10.1"
  ]
}

Or multiple indexed branches

{
  "url": "https://github.com/sbt/sbt.git",
  "indexedBranches": [
    "1.0.x",
    "0.13"
  ]
}

In which case the first branch in the list is taken to be the "primary" branch when the repository is first viewed, but the other branches are also indexed and searchable. Note you can also view and browse all branches whether or not they're indexed; this only affects whether a branch is indexed for searching.

Lastly, you can make your Fluent server mirror code from private repositories by giving it the path to a private key file with access to that repository. For example:

{
  "url": "git@github.com:lihaoyi/private.git",
  "gitPullSshKeyFile": ["id_rsa"]
}

Where id_rsa is a path (relative or absolute) to a key file with access to that repository.

You can also pass in a gitPullSshKeyPassPhrase if the given key file is protected by a passphrase:

{
  "url": "git@github.com:lihaoyi/private.git",
  "gitPullSshKeyFile": ["id_rsa"],
  "gitPullSshKeyPassPhrase": ["foopassword"]
}

You can generate the repos list via a script. For example, here is one that downloads a list of repositories from github and outputs it in the desired format.

#!/usr/bin/env python3

import requests, json

organization_name = "google"
access_token = "914caa34d39720a722949616daeb87cd4e723cac"
results = []
page = 0

while True:
    print(page)
    user_summaries = requests.get(
        "http://api.github.com/orgs/" + organization_name + "/repos",
        params = {"per_page": 100, "page": page},
        headers = {"Authorization": "token " + access_token}
    ).json()
    if user_summaries == []:
        break
    else:
        results += [
            {"url": x["url"], "indexedBranches": [x["default_branch"]]} 
            for x in user_summaries
        ]
        page += 1

print(json.dumps(results))

repoFolder

What folder to put the mirrored repositories in

indexCacheFolder

What folder to put the index files in

port/host

Port and host that fluent will bind to; defaults to localhost.

publicHost

What URL this Fluent server is available at publically, for use in email and single-sign-on. e.g. ["demo.fluentcode.com"]

smtpHost

What email server to use to send emails; defaults to localhost, which assumes you have an SMTP server running on your local machine (e.g. postfix). Could also be something like "smtp-relay.gmail.com" to use your Google Apps account's SMTP server

validUsers/validUsersCommand

Two ways to control who is able to login to this Fluent instance: either by a hardcoded list of users who can login, e.g.

"validUsers": {
  "email": ["foo@bar.com", "qux@bar.com"],
  "github": ["baz", "githubuserid"]
}

Which would allow users foo@bar.com and qux@bar.com to login through email/google login, and the github users baz and githubuserid to login through github login.

Or through a command that can be run:

"validUsersCommand": ["python3", "fetch_github_users.py"]

That will generate a JSON map in the same format at validUsers. This can be used to e.g. fetch users from github via a script:

#!/usr/bin/env python3

import requests, json

organization_name = "google"
access_token = "abcdef1234567890abcdef1234567890abcdef12"
results = []
page = 0

while True:
    user_summaries = requests.get(
        "http://api.github.com/orgs/" + organization_name + "/members",
        params = {"per_page": 100, "page": page},
        headers = {"Authorization": "token " + access_token}
    ).json()
    if user_summaries == []:
        break
    else:
        results += [x["login"] for x in user_summaries]
        page += 1

print(json.dumps({"github": results}))

Or to sync lists of emails from any other source of truth, such as an employee directory. validUsersCommand is run once on startup to load in the list of users, and then run once every 5 minutes after to keep the list of users in sync.

loginConfig

Allowed ways to log-in to your Fluent server. There are three options:

  • Email Login
  • Google Login
  • Github Login

You can use any combination of login methods in order to authenticate a user trying to use your Fluent server. You can use any one, or any combination of these authentication methods. After a user is authenticated using any one of the methods listed above, their email is retrieved and checked against either the validEmails list or emailValidatorCommand to see if the user is authorized to use this Fluent server.

Email Login

{"$type": "fluent.web.EmailLogin"}

This is the simplest login method; the Fluent server will send the user an email using the given smptHost and publicHost, containing a link that the user can click to log them into the Fluent server. To do this, you will need to first configure your smtpHost to point to a valid mail server, e.g. via a local Postfix server for testing or something like the Google Apps SMTP Relay for real usage.

Google Login

{
  "$type": "fluent.web.GoogleLogin",              
  "clientId": "123412341234-abcdefgabcdefgabcdefgabcdefgabcd.apps.googleusercontent.com",
  "clientSecret": "abcdef-abcedfghijklmnopq"
}

To enable single-sign-on with Google,

  • First create a project on the Google Developers Console: https://console.developers.google.com/

  • Add the URL of your fluent install to the Authorized Javascript origins, e.g. http://localhost:8080 for local testing or http://demo.fluentcode.com for a real deployment

  • Add the URL for the callback endpoint to the Authorized redirect URIs, e.g. http://localhost:8080/login/google/confirmation for local testing or http://demo.fluentcode.com/login/google/confirmation for a real deployment

  • Paste the given clientId and clientSecret into your fluent configuration file

Github Login

{
  "$type": "fluent.web.GithubLogin",
  "clientId": "abcedfabcdefabcdefab",
  "clientSecret": "abcdefabcdefabcdefabcdefabcdefabcdefabcd"
}

To enable single-sign-on with Github,

  • First create a Github OAuth Application via https://github.com/settings/developers

  • Set the Authorization Callback URL to e.g. http://localhost:8080/login/github/confirmation for local testing or http://demo.fluentcode.com/login/github/confirmation for a real deployment

  • Paste the given clientId and clientSecret into your fluent configuration file

loginPrimaryMessages/loginSecondaryMessages

Inspirational messages to appear on the login page every time users log in to your Fluent server. You can add a list of messages for each, and the displayed messages will be randomly chosen from these two lists each time the login page is served.