Skip to main content

Submission of GEOJSON data to the DCDB

All data submitted to the DCDB need to be in .geojson (preferred) or XYZ format and contain appropriate metadata fields. These requirements are outlined in DCDB guidance documents.

  • For For WIBL WIBL or or YDVR YDVR data, you will need to upload data periodically. See the WIBL and YDVR Data Processing protocol for full details.
  • For For Mussel Kit Kit data, OFM handles data submission on behalf of the Trusted Node, using the Trusted Node's secure token.
  • OFM can also provide the raw data back to Trusted Nodes via Dropbox or AWS S3 buckets (other options are also available).

Once your WIBL or YDVR data have been validated, they can be submitted to the DCDB using the following PowerShell script:

#Run #Run Windows PowerShell 7 as an administrator

#Run the following code
Set-ExecutionPolicy Unrestricted

#Close #Close Windows PowerShell 7, then reopen (not as admin)

#Set directory to WIBL data-management scripts folder
cd .\Desktop\WIBL-main\wibl-python\scripts\data-management\

#Activate environment
conda activate wibl-python

#Submit to DCDB
.\submitDCDB.ps1 [[-inPath] <string>] [[-authFile] <string>] [[-configFile] <string>] [[-extension] *.geojson]

submitDCDB.ps1: -inPath is the folder where validated .geojson files are saved, -authFile is the authentication token .txt file, -configFile should be the same as used for processWibl.ps1, and -extension should be set to *.geojson.

Example: .\submitDCDB.ps1 -inPath C:\Users\schernoch\Documents\CSB\Chance-Maritime\2-Validated\ -authFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\3-Submission\ingest-external-COMITUSF.txt -configFile C:\Users\schernoch\Documents\CSB\Chance-Maritime\0-Metadata\configure-submission-USF.json -extension *.geojson

When a file is successfully submitted to the DCDB, you should receive the following output:

image.png

Each time a file is submitted successfully, the program will print a submission ID. Make sure to log these submission IDs in case your files need to be tracked down by technical support staff at the DCDB for whatever reason.

If you are unable to batch-submit your processed data using the WIBL PowerShell script provided above, you can use either of the following options to manually submit your .geojson files to the DCDB

Option 1: Python script

Copy the following script into into Visual Studio Code  and replace <submissionToken_path.txt> and <geojson_path.geojson> with the appropriate file paths. Save the file as as submit_geojson.py py and make note of what folder you save it in.

import requests

token_path = r"<submissionToken_path.txt>"
geojson_path = r"<geojson_path.geojson>"
unique_id = "COMITUSF-e512gc2r-q565-279j-548i-e5fc6389cth2"
url = "https://www.ngdc.noaa.gov/ingest-external/upload/csb/geojson"

with open(token_path, "r") as f:
    token = f.read().strip()

headers = {
    "x-auth-token": token
}
files = {
    "file": ("wibl-raw.61.geojson", open(geojson_path, "rb"), "application/json"),
    "metadataInput": (None, f'{{"uniqueID": "{unique_id}"}}')
}

response = requests.post(url, headers=headers, files=files)

# === OUTPUT ===
print(f"Status Code: {response.status_code}")
print(response.text)

Perform the following steps to submit your .geojson file to the DCDB.

#Launch Command Prompt

#Call directory to the folder where submit_geojson.py is saved
cd .\<folder_path>

#Run the following code
python submit_geojson.py

#Record the submissionID

You should receive the following output if your file was successfully submitted to the DCDB:

image.png

Each time a file is submitted successfully, the program will print a submissionID. Make sure to log these submissionIDs in case your files need to be tracked down by technical support staff at the DCDB for whatever reason.  

If you have multiple .geojson files to submit, you will need to edit <geojson_path.geojson> within submit_geojson.py each time you submit a new file.

  • This can be done relatively quickly if you keep submit_geojson.py open within Visual Studio Code, making sure to save (ctrl+S) your changes before running the code again.
  • Once <geojson_path.geojson> has been edited to the new file path, you can click the up arrow in Command Prompt to reload the previous command and click enter to run it.

Options 2: Batch script

Perform the following steps to submit your .geojson file to the DCDB. Make sure to replace <submissionToken_path.txt> and <geojson_path.geojson> with the appropriate file paths.

#Copy and paste the following code (lines 3-17) into Visual Studio Code

@echo off
REM Define paths and values
set "TOKEN_FILE=<submissionToken_path.txt>"
set "GEOJSON_FILE=<geojson_path.geojson>"
set "UNIQUE_ID=COMITUSF-e512gc2r-q565-279j-548i-e5fc6389cth2"

REM Read token
set /p TOKEN=<"%TOKEN_FILE%"

REM Run curl
curl -i -X POST "https://www.ngdc.noaa.gov/ingest-external/upload/csb/geojson"^
 -H "x-auth-token: %TOKEN%"^
 -H "Content-Type: multipart/form-data"^
 -F "file=@%GEOJSON_FILE%;type=application/json"^
 -F "metadataInput={\"uniqueID\":\"%UNIQUE_ID%\"}"
 
 #Replace <submissionToken_path.txt> and <geojson_path.geojson> with appropriate file paths
 
 #Launch Command Prompt
 
 #Copy and paste your edited code into Command Prompt
 
 #Run the code
 
 #Record the submissionID

You should receive the following output if your file was successfully submitted to the DCDB:

image.png

If you have multiple .geojson files to submit, each time you submit a new file you will need to edit <geojson_path.geojson> in Visual Studio Code and then run the edited code in Command Prompt.

Recommended: Test submission

If you would like to test out running the scripts without posting data to the DCDB Data Viewer, simply add /test/ to the endpoint URL, so it instead reads: https://www.ngdc.noaa.gov/ingest-external/upload/csb/test/geojson

The test endpoint should always be used first for any system or script testing. Submissions sent to the DCDB production endpoint are considered final submissions and are subject to a lengthy process to remove if submitted in error.

Back to top