Ingest External Data Feeds

Before beginning this procedure, you'll need the URL to an open-source data feed.

Process

  1. Open the open-source data feed.

  2. Identify the column heads and separation symbol.

    Column Heads: Separation Symbol:

    Firstseen (UTC),
    Threat,
    Malware,
    Host,
    URL,
    Status,
    Registrar,
    IP address(es),
    ASN(s),
    Country

    		 comma
  1. In your favorite text-edit application, update spec.xml to contain the column heads you need.

Still in your text-edit application, create a sample_data.txt file that contains one or more rows of data from the feed.

  1. Log in to Asset Manager Command Center using your browser interface.
  2. Go to Settings > Tables > Add Table.
  3. In the Name field, enter a descriptive name for the table you are creating such as ransomware_tracker_feed.
  4. In the Table Type field, select Managed Primary Table.
  5. At your option, you can add Tags to help others find the table and a Description for the purpose of the table.
  6. Browse to and then select the spec.xml and sample_data.txt files.
  7. Click Next.
    The column headings for your table display, ordered alphabetically. Review the values listed in the Field column to confirm that they match your spec.xml. 
  8. Click the Value Index checkbox for those fields you want to HDFS to index immediately. If you do not check any items in the Value Index column, your table with still be created, but not indexed.  
  9. Click Create Table.
    The table structure is created in the Asset Manager's HDFS data store.
  1. Log in to the CLI of your Asset Manager Command Center.
  2. At the command-line prompt, enter support db.
    You now have access to Asset Manager's PostgreSQL database.
  3. Insert the feed details into Asset Manager's PostgreSQL database using the Insert command.  The labels will remain the same from feed to feed. The values for each label will need to be customizer for your feed. Here's a sample entry:
    insert into system.feed(name, shortname, enabled, overwrite, url, key, filename, tablename, pipelinename, pollinterval) values ('ransomware-tracker', 'ransomware', true, true, 'https://ransomwaretracker.abuse.ch', '','/feeds/csv/', 'Asset Manager.public.ransomware_tracker_feed', 'ransomware',1440);
  1. Make sure the insertion was received by entering
    select * from system.feed where name ='ransomware-tracker';
    A response indicates that Asset Manager's database has received the insertion.
  2. To validate the connection, restart the Asset Manager API service by entering:
    support service apirestart
     The feed will begin to populate and records will very soon be available in the Asset Manager GUI.
  1. In the Asset Manager, go to Settings > Tables.
  2. Select the ransomware_tracker_feed table.
  3. Check the number of records present to confirm that the database has been populated.
  4. Click View.
    The table displays. Asset Manager has ingested an external data feed.