Your own server

Is it for you?

A private server is the same Collect Earth Cloud that FAO runs at cloud.collectearth.org: the web dashboard, the sync service Collect Earth talks to, and a database with nightly backups. Coordinators and interpreters work exactly as on the shared server. The difference is where it runs and who controls it.

• The data stays with you. Everything runs in your organisation's own Google Cloud project, or on your own hardware. Nobody else, including FAO and the Collect Earth team, can see it, and the server sends no usage data anywhere unless you switch it on.
• You decide the rules: who gets an account, who may create projects, how much each project may hold, and which projects exist at all.
• No programming needed. On Google Cloud a guided panel and one command do the work. You need someone comfortable with a web console and copying a command, and a payment card on a Google Cloud billing account.

If your data may be kept on FAO's server, the free Collect Earth Cloud needs no installation at all. You can also start there and move later (see Moving a project to your server).

What it costs

You pay Google for what the server uses, every month, on your billing account. Almost all of it is the database, which runs day and night; the server itself runs only while someone uses it, and usually stays within Google's monthly free allowance.

Where the 10 US$ go

By region and size

The database machine is the choice that matters most: the default db-f1-micro suits one or a few projects; a busier server needs a bigger one, chosen when the database is created (see below). Prices also differ between regions.

US$ a month. The install script offers the regions without an asterisk; the others, and any region with Cloud Run and Cloud SQL, work with --region. Choose the region nearest to the people who will use the server: the cheapest regions are usually close enough for a team on another continent.

Optional extras

What is free

• Cloud Run, each month: 180,000 vCPU-seconds, 360,000 GiB-seconds and 2 million requests. These allowances belong to the billing account, so other projects on the same account share them.
• Cloud Build: 2,500 build-minutes a month (an install or update takes 3 to 6). Secret Manager: 6 secret versions and 10,000 reads (the server uses 3). Artifact Registry: 0.5 GB. Cloud Scheduler: 3 jobs.
• New Google Cloud customers get a 300 US$ credit to spend over 90 days, which covers the first months.

Keeping the bill low

• Start with the smallest database and move up only if pages become slow. You can change it later (see A bigger database).
• Keep the server's default request-based billing: a quiet server then costs almost nothing beyond the database.
• Set a budget alert on the billing account, a few dollars above what you expect. It sends an email; it doesn't stop anything.
• For a server that runs for years on a dedicated database, a one-year committed use discount lowers the database machine by 25 % (for example 49 to 37 US$ a month). It does not apply to the shared-CPU machines.
• When the work is over, download the data and delete the project (see Removing it).

Before you start

1. A Google account. Any Gmail or Google Workspace account. If your organisation uses Google Workspace, use your work account, so that the server belongs to the organisation and not to a person.
2. Google Cloud. Open console.cloud.google.com with that account and accept the terms the first time.
3. A billing account: how Google charges you, with a payment card. Create one at console.cloud.google.com/billing, or ask whoever manages Google Cloud in your organisation to link theirs to your project.
4. A new project, just for this server, so its cost is easy to see and everything can be deleted at once. The install panel helps you create it. You need the Owner role on it.
5. Choose the region, the place where the server will run: the one nearest to the people who will use it, keeping the price differences in mind. It cannot be changed afterwards without moving the data.

Install it on Google Cloud

Press the button. It opens Cloud Shell, a terminal in your browser provided by Google for free and already signed in with your account, with the Collect Earth code and a step-by-step panel on the right.

Open in Cloud Shell

1. Trust the repository. Cloud Shell asks whether you trust it: tick Trust repo and press Confirm. Without this the terminal has no access to your Google account and the installation cannot work.
2. Pick the project in the panel on the right (or create one there), and check that billing is on. The panel shows a command that tells the terminal which project to use: copy it and press Enter.
3. Authorize. When the first command runs, Google may ask you to let the terminal act with your account: press Authorize.
4. Run the installation:

   bash deploy/gcp/deploy.sh

5. Choose the region from the list (Belgium, Iowa, South Carolina, São Paulo, Johannesburg, Mumbai or Singapore), or press Enter for Belgium. Any other region works with --region, for example bash deploy/gcp/deploy.sh --region asia-southeast2. The script then shows what it will do and asks Continue?: press Enter.
6. Wait. It works through 8 steps and prints each one: billing, Google services, the database (10 to 15 minutes), the passwords and setup code, the server's own account, the place for server versions, building the server (3 to 6 minutes) and starting it. Keep the tab open; you can do something else meanwhile.
7. Write down the two things it prints at the end: the dashboard address, like https://collect-earth-cloud-123456789.europe-west1.run.app, and the setup code, like K7QX-M3TB-WN.

What it creates

Everything is in your project, with names starting with ce:

Every command is listed in DEPLOY.md. To see them without changing anything, run bash deploy/gcp/deploy.sh --dry-run.

First steps on your server

1. Create the administrator. Open the dashboard address. The setup page asks for the setup code (capitals, spaces and dashes don't matter) and for the user name and password of the server administrator: you. The code works only once; afterwards you log in with that name and password. Lost the code before using it? bash deploy/gcp/deploy.sh --show-setup-code shows it again.
2. Server settings. Open Server. Leave Anyone can create an account off, so people only get an account through an invitation to a project. Choose Who can create projects: only server administrators, or everyone who has an account. You can also give the server a name, a contact email shown on every page, and limits per account.
3. Your first project. My projects › New project, then follow the checklist: the project file, the plot files, invitations, who collects each file. From here on it works exactly as in Running a cloud project.
4. Invite the team. Each invitation link carries your server's address, so interpreters never type it: they choose File › Join cloud project... in Collect Earth and paste the link.

The server administrator sees no project's data unless they are also a member of the project: their Server page shows totals only, with projects by number.

Optional: address, emails, AI, size

Your own address

Google's address works and is secure. For an address of your organisation, such as https://cloud.example.org, prove once that you own the domain with gcloud domains verify example.org, then run:

bash deploy/gcp/deploy.sh --domain cloud.example.org

The script prints the DNS records to add where the domain was bought. This works in Belgium, Iowa, South Carolina and Singapore among others; in other regions, such as Johannesburg, Mumbai or São Paulo, the script says so, and a load balancer or another proxy is needed in front (DEPLOY.md, "Your own address").

Emails

Without email the server works fully; you make password reset links yourself (Server › Accounts › Make a reset link). With a mail provider that sends for your domain, people can use Forgot your password? and ask for a weekly summary. You give the server the provider's details (SMTP_URL), a sender address, the server's address and a signing secret; the commands are in DEPLOY.md, "Weekly emails (optional)".

The AI assistant

Off unless you set it up. It needs the Vertex AI service turned on in your project, then Allow the AI assistant on this server in the server settings, and each coordinator who wants it turns it on in their project. It sends Google's Gemini the choice and number answers, the questions and the quality figures, never coordinates, plot ids, comments or names. Limits per project (by default 5 reviews a day and 2 million tokens a month) keep the cost small. Steps: DEPLOY.md, "AI assistant (optional)".

A bigger database, more memory

• For a new server, choose the machine when you install: bash deploy/gcp/deploy.sh --db-tier db-g1-small (1.7 GB of memory) or --db-tier db-custom-1-3840 (1 dedicated vCPU, 3.75 GB).
• For an existing server: gcloud sql instances patch ce-cloud-db --tier=db-g1-small. The database restarts, which takes a few minutes; Collect Earth keeps the plots and sends them afterwards.
• Large projects (above about 30,000 plots) need more memory for the server too: MEMORY=1Gi bash deploy/gcp/deploy.sh, with the settings listed in DEPLOY.md, "Memory".

Looking after it

Updating

When there is a new version, press Open in Cloud Shell again and run the same command, bash deploy/gcp/deploy.sh. It fetches the latest code, rebuilds the server and switches to it in about 5 minutes, without asking the questions again. The data stays: the database is never recreated, and people working at that moment don't notice.

Backups and restoring

The database is backed up every night, the last 7 copies are kept, and it can be put back as it was at any minute of the last 7 days. To restore, open the Cloud SQL console, the database ce-cloud-db, then Backups (Google's guide). Restoring replaces all current data, so download each project's data first. Try a restore once before the real work starts, so you know it works.

For a copy outside Google Cloud, download each project's data from its Data tab now and then: a spreadsheet, or a Collect Earth backup that can start the project again anywhere.

Security checklist

• Keep Anyone can create an account off for a private server.
• Give the Owner role on the Google Cloud project to as few people as possible.
• Mind who can read the logs. Google's request log records full addresses, including invitation, progress and password-reset links. Give log access only to people who may see the projects, or exclude those addresses from the log (DEPLOY.md, "Operational notes").
• Keep the database's deletion protection on, and the budget alert set.
• Update when a new version comes out.

When something goes wrong

The server's own log shows what it did (use your region):

gcloud run services logs read collect-earth-cloud --region europe-west1 --limit 50

Ask in the Open Foris user forum or the Collect Earth issues, with the messages from the terminal and the dashboard address.

Moving a project to your server

A project that started on cloud.collectearth.org, or on a computer, moves with its survey and its plots:

1. On the old project's Data tab, take a Collect Earth backup. From a computer, export XML (compressed) instead.
2. On your server, My projects › New project › From a Collect Earth backup. The survey and the plots already assessed come in; those plots go into a plot file of their own, ending in _done.csv, to leave out when you give out plots.
3. Add the plot files for the remaining work, invite the team again, and give out the plots. Interpreters join the new project with its link, as the first time.
4. When everything is there, delete the old project (its Settings tab).

On any other host, with Docker

Collect Earth Cloud is one container and a PostgreSQL database, so it also runs on your own servers or any other cloud. This route is for a system administrator.

What you need

• A machine with Docker, reachable from the internet.
• PostgreSQL 16 or later, with backups: a managed database or your own, with a database collect_earth_cloud and an owner role (here ce). The server creates the less powerful role it runs as (here ce_app) itself, the first time it starts.
• HTTPS in front: the server speaks plain HTTP on port 8080, so put a load balancer or a reverse proxy, such as Caddy or nginx, in front of it.

Install

1. Build the image:

   git clone https://github.com/openforis/collect-earth
   cd collect-earth/collect-earth-cloud
   docker build -t collect-earth-cloud .

2. Run the server. Choose a setup code, such as K7QX-M3TB-WN, or leave SETUP_CODE out and read the one the server prints in its log (docker logs ce-cloud):

   docker run -d --name ce-cloud --restart unless-stopped -p 127.0.0.1:8080:8080 \
     -e STORE_BACKEND=postgres \
     -e MIGRATION_DATABASE_URL='postgres://ce:OWNER_PASSWORD@db-host:5432/collect_earth_cloud' \
     -e DATABASE_URL='postgres://ce_app:APP_PASSWORD@db-host:5432/collect_earth_cloud' \
     -e MIGRATE_ON_BOOT=true \
     -e SETUP_CODE='YOUR-SETUP-CODE' \
     -e PUBLIC_BASE_URL='https://cloud.example.org' \
     collect-earth-cloud

3. Put HTTPS in front. With Caddy, which gets the certificate by itself, the whole configuration is:

   cloud.example.org {
       reverse_proxy 127.0.0.1:8080
   }

   Behind two proxies, such as a load balancer in front of Caddy, add -e TRUST_PROXY_HOPS=2.
4. Check it: https://cloud.example.org/health answers, and /v1/server-info says needsSetup until you have created the administrator. Then continue with First steps on your server.

To update: fetch the code again (git pull), build the image, and replace the container with the same docker run; the database is updated when the server starts. Every setting, including email and the AI assistant, is in DEPLOY.md. The docker-compose.yml in the folder is a database for trying things out only, not for real work.

Removing it

To stop all costs on Google Cloud, delete the project. This deletes the server and the database with all the data collected, so download each project's data first.

1. Open Manage resources, select the project, press Delete and type its ID. Or, in Cloud Shell: gcloud projects delete YOUR-PROJECT-ID.
2. Billing stops at once. Google keeps the project for 30 days in case you change your mind, then removes it for good.