Skip to content

Content Node: Architecture

Jump to bottom
Sid Sethi edited this page Oct 29, 2020 · 1 revision

What is an Audius Content Node?

An Audius Content Node maintains the availability of creators' content on IPFS. The information stored includes Audius user metadata, images, and audio content.

NOTE - Previously, there was a concept of a ”creator node” that was separate from a content node. These have been combined into a single node type rather than being separate, with ”content node” referring to the merged type. Some references to ”creator node” still exist in Audius code and in other documentation; those can safely be assumed to be referring to the content nodes outlined here.

Table of Contents

Architecture

The content node is written in NodeJS. The node uses FFMPEG to process & segment tracks.

Web Server

The content node's core service is a web server with an HTTP API to process incoming requests and perform the following functions:

  • user & track metadata upload
  • user & track image upload
  • user track file upload
  • user & track associations with ETH contract values
    • All users and tracks are tied to Ethereum wallet addresses and unique IDs
  • user & track data, metadata, and file retrieval

The web server is an Express app.

IPFS Node

A content node runs a dedicated IPFS node to provide decentralized availability of all Audius creator content. The web server interacts with an IPFS node through the JS IPFS API

Persistent Storage (Postgres)

It stores all data in a SQL database and all images and metadata objects on its IPFS node. Each track is stored twice, once in persistent storage and second on a dedicated IPFS node, which is treated as ephemeral.

Pointers to all content and metadata stored in IPFS are persisted in the Postgres DB.

Postgres is managed in the codebase using the Sequelize ORM which includes migrations, models and validations

Redis

A Redis client is used for for resource locking, request rate limiting, and limited caching and key storage.

Redis is managed in the codebase through the ioredis npm package

Track Segmenting

As defined by the Audius protocol, the content node segments all uploaded track files before storing/serving.

Data Redundancy

As defined by the Audius protocol, all creators are encouraged to store content redundantly across multiple nodes to maximize availability. Each node thus exposes sync functionality that a creator can use to transfer their metadata and files to another node. Metadata is transferred directly and files are transferred directly through the peer IPFS nodes.

Persistent storage can be cloud-backed (e.g. S3) or a local volume / directory.

Data Model

CNodeUser

A CNodeUser object represents a content node user, created for each Ethereum owner wallet address. All other data models are associated with a wallet indirectly through a CNodeUser.

AudiusUser

An AudiusUser object represents an Audius user on the Ethereum blockchain. It references Tracks for all track entries, references Files for coverArt, profilePic, and user metadata object, and redundantly holds the user metadata JSON object. Each AudiusUser is owned by a CNodeUser.

Track

A Track object represents an Audius track on the Ethereum blockchain. It references Files for coverArt and track metadata object, and redundantly holds the track metadata JSON object. Each Track is owned by an AudiusUser.

File

A File object holds the CID of the file pinned by the IPFS node, the name of the uploaded source file, and the file's storage path in the content node's persistent storage. Each File is owned by a Track or AudiusUser.

SessionToken

A SessionToken object allows content node clients permissioned access to their own data on that node, and blocks access to other CNodeUser's data. Each SessionToken is owned by a CNodeUser.

All content node objects are identified with UUIDs to account for the distributed nature of this data, allowing safe data transfer between nodes.

API

CNodeUser

  • POST /users

    Authorize a new CNodeUser to use the content node, and enable all other data creation.

  • [DEPRECATED] POST /users/login

    Given wallet address and valid signature for key associated with wallet, and creates session for future interaction with node.

  • GET /users/login/challenge

    Given the public wallet key, returns a challenge for validating user login and sets the challenge in Redis cache

  • POST /users/login/challenge

    Checks if challenge in request body matches up with what we have stored. If request challenge matches what we have, remove instance from Redis to prevent replay attacks. Return sessionToken upon success.

  • POST /users/logout

    Given token, deletes node session for wallet.

AudiusUser

  • POST /audius_users/metadata

    Create AudiusUser from provided metadata, and make metadata available to network.

  • POST /audius_users/associate/{audiusUserUUID}

    Given audiusUser blockchainUserId, blockNumber, and metadataFileUUID, creates/updates AudiusUser DB entry and associates image file entries with audiusUser. Ends audiusUser creation/update process.

Track

  • POST /track_content

    Given uploaded track file, segments it into multiple files, stores each file in persistent storage, adds to IPFS node, and creates DB File entry. Track will be associated as Audius Track in POST /tracks route.

  • POST /tracks/metadata

    Given track metadata object, upload and share metadata to IPFS. Return metadata multihash if successful. Error if associated track segments have not already been created and stored.

  • POST /tracks

    Given track blockchainTrackId, blockNumber, and metadataFileUUID, creates/updates Track DB track entry and associates segment & image file entries with track. Ends track creation/update process.

  • GET /tracks/download_status/{blockchainId}

    Returns download status of track and 320kbps CID if ready + downloadable.

  • GET /tracks/stream/:encodedId

    Gets a streamable mp3 link for a track by encodedId. Supports range request headers.

  • GET /tracks/unlisted

    List all unlisted tracks for a user

File

  • POST /image_upload

    Given image file, save multiple-resolutions in persistent storage and to IPFS, then create DB File entries.

  • GET /ipfs_peer_info

    Returns data pertaining to IPFS client

  • GET /ipfs/{CID}

    Given IPFS CID, returns file stream if stored on node via file system. If not available on file system, get and stream content from IPFS.

  • GET /ipfs/{dirCID}/{filename}

    Gets a CID in a directory, streaming from the filesystem if available and falling back to IPFS if not. The filename is one of the multi-resolution images served.

NodeSync

  • GET /export

    Given wallet array, returns all DB data including CNodeUser, AudiusUsers, Tracks, Files. Also rehydrates all files in IPFS nodes per cnode UUID if the files associated with that cnode UUID has not been rehydrated in the past hour.

  • POST /sync

    Given wallet array and target content node endpoint, fetches data export from endpoint for wallets, then stores all fetched data in DB. Peers to other node's IPFS node and stores in persistent storage.

  • GET /sync_status/{walletPublicKey}

    Checks if node sync is in progress for a wallet

HealthCheck

  • GET /health_check

    Returns 200 with info pertaining to the current version, current service, selected Discovery Node, and if the web server and the db connection are both up

  • GET /version

    Returns content node version data required for Audius protocol participation.

  • GET /health_check/ipfs

    Performs diagnostic ipfs operations to confirm functionality

  • GET /db_check

    Exposes current and max db connection stats. Returns error if db connection threshold exceeded, else success.

  • GET /disk_check

    Exposes current and max disk usage stats. Returns error if max disk usage exceeded, else success.

Usage Example Flows

Track Creation Flow

  1. POST /track_content Client uploads track file to content node
    1. Receives back segment multihash array as proof of storage / availability
  2. Client packages returned multihash array into track metadata object
  3. POST /tracks Client uploads track metadata object to content node
    1. Receives back metadata multihash as proof of storage / availability
  4. Client uploads track metadata object to Audius blockchain
    1. Receives back blockchain track ID as proof of record
  5. [POST /tracks/associate/:id] Client uploads blockchain track ID to content node to prove blockchain upload of track
    1. Content node associates blockchain track ID with previously created track
    2. this completes track creation process

Track Update Flow

  1. Client posts any image files to [POST /image_upload] if image was changed and updates multihash in metadata object
  2. [POST /metadata] Client uploads the new metadata to content node and retrieves multihash
  3. Client updates multihash on chain for track
    1. Receives the transaction receipt to confirm success transaction
  4. [PUT /track/:id] Client put's the metadata to the track update endpoint, which will update the content node db and make any associated changes now to mirror the blockchain
Clone this wiki locally