Ingesting External Data Feeds

Before beginning this procedure, you'll need the URL to an open-source data feed. Illustrations on this page use to serve as a representative example.


  1. Open the open-source data feed.
  2. Identify the column heads and separation symbol.

    Column Heads: Separation Symbol:

    Firstseen (UTC),
    IP address(es),

  3. In your favorite text-edit application, update spec.xml to contain the column heads you need. Your updated xml file should look similar to this:

Still in your text-edit application, create a sample_data.txt file like this one that contains one or more rows of data from the feed.

  1. Log in to Spectre Command Center via your browser interface.
  2.  Navigate to Settings >Tables > Add Table.
  3. In the Name field, enter a descriptive name for the table you are creating such as ransomware_tracker_feed.
  4. In the Table Type field, select Managed Primary Table.
  5. At your option, you can add Tags to help other Spectre users to find the table and a Description to let others know the purpose of the table.
  6. Browse to and then select the spec.xml and sample_data.txt files.
  7. Click Next.
    The column headings for your table display, ordered alphabetically. Review the values listed in the Field column to confirm that they match your spec.xml. 
  8. Click the Value Index checkbox for those fields you want to HDFS to index immediately. If you do not check any items in the Value Index column, your table with still be created–just not indexed.
  9. Click Create Table.
    The table structure is created in the Spectre's HDFS data store.
  1. Log in to the CLI of your Spectre Command Center.
  2. At the command-line prompt, enter support db.
    You now have access to Spectre's PostgreSQL database.
  3. Insert the feed details into Spectre's PostgreSQL database using the Insert command.  The labels will remain the same from feed to feed. The values for each label will need to be customizer for your feed. Here's a sample entry:
    insert into system.feed(name, shortname, enabled, overwrite, url, key, filename, tablename, pipelinename, pollinterval) values ('ransomware-tracker', 'ransomware', true, true, '', '','/feeds/csv/', 'Asset Manager.public.ransomware_tracker_feed', 'ransomware',1440);
  4. Make sure the insertion was received by entering
    select * from system.feed where name ='ransomware-tracker';
    A response similar to this one indicates that Spectre's database has received the insertion.
  5. To validate the connection, restart the Asset Manager-api service by entering:
    support service apirestart

     The feed will begin to populate and records will very soon be available in the Spectre GUI.
  1. In the Spectre GUI, browse to Settings > Tables.
  2. Select the ransomware_tracker_feed table.
  3. Check the number of records present to confirm that the database has been populated.
  4. Click View.

    Congratulations! The table displays. Spectre has ingested an external data feed.

Congratulations! You have caused Spectre to ingest a feed of external data.