FRANKLY

BLOG

Building a StockX API Integration: Fetching Market Data for Yeezy Sneakers

Aug 7, 2024

In this blog post, we'll explore the development of a Node.js application that integrates with the StockX API to fetch market data for Yeezy sneakers. This app demonstrates how to authenticate with the StockX API, process CSV files, and retrieve product market data programmatically.

Project Overview

Our application performs the following tasks:

Authenticates with the StockX API using OAuth 2.0
Reads product information from a CSV file
Fetches market data for each product from the StockX API
Updates the CSV file with the retrieved market data

Let's dive into the key components of this application.

Authentication Flow

The StockX API uses OAuth 2.0 for authentication. Our app implements this flow in the following steps:

Open a browser window for user authorization
Receive the authorization code via a callback
Exchange the authorization code for an access token

Here's a snippet of the authentication process:

};

const server = https.createServer(options, app).listen(3000, () => {
  console.log('Listening on https://localhost:3000');
});

async function getAuthorizationCode() {
  const authUrl = new URL('https://accounts.stockx.com/authorize');
  authUrl.searchParams.append('response_type', 'code');
  authUrl.searchParams.append('client_id', CLIENT_ID);
  authUrl.searchParams.append('redirect_uri', REDIRECT_URI);
  authUrl.searchParams.append('scope', 'offline_access openid');
  authUrl.searchParams.append('audience', 'gateway.stockx.com');
  authUrl.searchParams.append('state', 'abcXYZ9876');

  console.log('Authorization URL:', authUrl.toString());
  console.log('Opening browser for StockX authentication...');
  try {
    await open(authUrl.toString());
    console.log('Browser opened successfully. Please log in to StockX.');
  } catch (error) {
    console.error('Error opening browser:', error);
    console.log('Please manually open the authorization URL in your browser.');
  }

  return new Promise((resolve, reject) => {
    const checkCode = setInterval(() => {
      if (authorizationCode) {
        clearInterval(checkCode);
        clearTimeout(timeout);
        resolve(authorizationCode);
      }
    }, 1000);

    const timeout = setTimeout(() => {
      clearInterval(checkCode);
      reject(new Error('Authorization timed out after 2 minutes'));
    }, 120000); // 2 minutes timeout
  });
}

This code initiates the OAuth flow, opens a browser for user authorization, and exchanges the received code for an access token.

Processing CSV Files

The app reads from and writes to CSV files using the csv-parser and csv-writer libraries. Here's how we process the input CSV:

async function processCSV() {
  console.log("Note: The 'lowestAskAmount' is not provided in the API response. This column will be kept in the CSV but left blank.");
  
  const results = [];
  const inputFile = 'yeezy_updated.csv';
  const outputFile = 'yeezy_updated_with_market_data.csv';

  // Read the input CSV file
  await new Promise((resolve, reject) => {
    fs.createReadStream(inputFile)
      .pipe(csv())
      .on('data', (data) => results.push(data))
      .on('end', resolve)
      .on('error', reject);
  });

  console.log(`Read ${results.length} rows from ${inputFile}`);
  console.log('CSV Headers:', Object.keys(results[0]));

  // Get unique productIds that don't already have market data
  const uniqueProductIds = [...new Set(results
    .filter(row => !row.variantId) // Skip rows that already have market data
    .map(row => row.productId))];
  console.log(`Found ${uniqueProductIds.length} unique product IDs without market data`);

  const totalProducts = uniqueProductIds.length;
  let processedProducts = 0;
  const marketDataMap = new Map();

  for (const productId of uniqueProductIds) {
    try {
      console.log(`Processing product ID ${++processedProducts}/${totalProducts}: ${productId}`);
      const marketData = await getProductMarketData(productId);
      if (marketData) {
        marketDataMap.set(productId, marketData);
        console.log(`Received market data for Product ID ${productId}`);
      } else {
        console.log(`No market data available for Product ID ${productId}`);
      }
    } catch (error) {
      console.error(`Error processing Product ID ${productId}:`, error);
    }

    // Add a delay between requests to avoid rate limiting
    await new Promise(resolve => setTimeout(resolve, 1000));
  }

This function reads the input CSV, extracts unique product IDs, and prepares to fetch market data for each product.

Fetching Market Data

The core functionality of our app is fetching market data from the StockX API. We implement this in the getProductMarketData function:

async function getProductMarketData(productId, retryCount = 0) {
  if (Date.now() >= tokenExpirationTime) {
    await refreshAccessToken();
  }

  const url = `https://api.stockx.com/v2/products/${productId}/market-data?currency=GBP`;

  try {
    const response = await fetch(url, {
      method: 'GET',
      headers: {
        'x-api-key': API_KEY,
        'Authorization': `Bearer ${accessToken}`,
        'Content-Type': 'application/json',
      },
      agent: new https.Agent({ rejectUnauthorized: false }) // For local development only
    });

    if (response.status === 403) {
      if (retryCount < 3) {
        console.log(`Received 403 for ProductId ${productId}. Refreshing token and retrying...`);
        await refreshAccessToken();
        return getProductMarketData(productId, retryCount + 1);
      } else {
        throw new Error(`Max retries reached for ProductId ${productId}`);
      }
    }

    if (response.status === 429) {
      const retryAfter = response.headers.get('Retry-After') || 60;
      console.log(`Rate limited. Waiting for ${retryAfter} seconds before retrying...`);
      await new Promise(resolve => setTimeout(resolve, retryAfter * 1000));
      return getProductMarketData(productId, retryCount);
    }

    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }

    const data = await response.json();
    return data;
  } catch (error) {
    console.error(`Error fetching market data for ProductId ${productId}:`, error);
    return null;
  }
}

This function makes an API request to StockX for each product ID, handling rate limiting and token refreshing as needed.

Updating the CSV with Market Data

After fetching the market data, we update our CSV file with the new information:

async function updateYeezyCSVWithMarketData() {
  const results = [];
  return new Promise((resolve, reject) => {
    fs.createReadStream('yeezy_updated.csv')
      .pipe(csv())
      .on('data', (row) => {
        results.push(row);
      })
      .on('end', async () => {
        console.log(`Read ${results.length} rows from yeezy_updated.csv`);
        
        for (let i = 0; i < results.length; i++) {
          const row = results[i];
          if (row.productId) {
            console.log(`Fetching market data for ProductId ${row.productId} (${i + 1}/${results.length})`);
            const marketData = await getProductMarketData(row.productId);
            if (marketData && marketData.variants && marketData.variants.length > 0) {
              const variant = marketData.variants.find(v => v.size === row.Size) || marketData.variants[0];
              row.variantId = variant.id || '';
              row.currencyCode = 'GBP';
              row.lowestAskAmount = variant.market.lowestAsk || '';
              row.highestBidAmount = variant.market.highestBid || '';
              row.sellFasterAmount = variant.market.sellFaster || '';
              row.earnMoreAmount = variant.market.earnMore || '';
            }
            await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay between API calls
          }
        }
        
        const csvWriter = createCsvWriter({
          path: 'yeezy_final.csv',
          header: Object.keys(results[0]).map(key => ({ id: key, title: key }))
        });
        
        await csvWriter.writeRecords(results);
        console.log('yeezy_final.csv has been created with market data added');
        resolve();
      })
      .on('error', (error) => {
        console.error('Error processing yeezy_updated.csv:', error);
        reject(error);
      });
  });
}

This code writes the updated data back to a new CSV file, including the newly fetched market data.

Running the Application

The main execution flow of our application looks like this:

(async () => {
  try {
    console.log('Starting authorization process...');
    const code = await getAuthorizationCode();
    globalAccessToken = await getAccessToken(code);
    console.log('Authorization completed successfully');
    
    console.log('Starting CSV processing...');
    await processCSV();
    console.log('CSV processing completed successfully');
  } catch (error) {
    console.error('An error occurred:', error);
  } finally {
    server.close(() => {
      console.log('Server closed');
      process.exit();
    });
  }
})();

This code initiates the authorization process, processes the CSV, and handles any errors that may occur during execution.

Conclusion

Building this StockX API integration demonstrates several important concepts in API development:

Implementing OAuth 2.0 authentication
Handling rate limiting and token refreshing
Processing CSV files in Node.js
Making asynchronous API calls in a loop
Error handling and logging in a Node.js application

This project provides a solid foundation for building more complex applications that interact with the StockX API or similar e-commerce platforms. By following these patterns, developers can create robust integrations that respect API limitations while efficiently processing large datasets.

Remember to always review and comply with the API provider's terms of service and usage guidelines when building integrations like this one.

Change theme