Skip to content

Instantly share code, notes, and snippets.

@jamiephan

jamiephan/README.md

Last active September 20, 2024 11:45
Show Gist options
  • Save jamiephan/0c04986c7f2e62d5c87c4e8c8ce115fc to your computer and use it in GitHub Desktop.
Save jamiephan/0c04986c7f2e62d5c87c4e8c8ce115fc to your computer and use it in GitHub Desktop.
Download ZIP
A script to automatically add ALL items to your account in quixel
Raw
README.md

Script to add all items from quixel

As quixel is being removed, all items are free to aquire. This script is to automate the process to add items to your account (As of writing, a total of 18874 items)

Note: This script only tested in the latest version of Chrome.

How to use

  1. Copy the script from below (run.js)
  2. Login into https://quixel.com
  3. Go to https://quixel.com/megascans/collections
  4. Open devtools (F12) -> Go to "Console" tab
  5. Paste in the script and press Enter.
  6. A dialog should popup confirming the execution, click "OK"
  7. Sit back and wait

Common issues

  • Getting "Forbidden" error. (Even after refresh, the whole page just shows "Forbidden")
    • There is a chance that the API adding too fast and you hit the rate limit of the API. (My testing is around after 10 pages, so ~10k items).
    • Wait after ~10-20 minutes and continue. See Common Fixes -> Restart script to continue the execution after you can load https://quixel.com.
  • The script seems to be paused/hang
    • It could be too much logging going it. Try monitor the script, if it says "END PAGE X", note the page number down (in case need restart) and clear the console by clicking the "🚫" icon in devtools.
    • See Common Fixes -> Restart script for fixing.
  • Getting the error **UNABLE TO ADD ITEM**
    • There should have the error message shown in ( ). If it is user already owns specified asset at a higher or equal resolution, then its already in your account.
  • Getting the error cannot find authentication token. Please login again
    • Clear browser cookies and re-login quixel again. Try just simply add 1 item manully. If it success, then see Common Fixes -> Restart script for continue the execution.

Common Fixes

Restart Script

  1. Note which page it was running
  2. Copy the run.js script
  3. Update the startPage = 0 on the first line to startPage = 10 (assuming page 10 was hanged)

Change Log

  1. Initial Script launch
  2. Update to clear logs to reduce chance of hanging
  3. [CURRENT] Skip adding items that already was acquired. Reduced logs. Added more info after script completion to show purchased item count. Due to now skipping purchased items, you technically don't need to specify the startPage anymore.
Raw
run.js
((async (startPage = 0, autoClearConsole = true) => {
const getCookie = (name) => {
const value = `; ${document.cookie}`;
const parts = value.split(`; ${name}=`);
if (parts.length === 2) return parts.pop().split(';').shift();
}
const callCacheApi = async (params = {}) => {
const defaultParams = {
page: 0,
maxValuesPerFacet: 1000,
hitsPerPage: 1000,
attributesToRetrieve: ["id", "name"].join(",")
}
const response = await fetch("https://proxy-algolia-prod.quixel.com/algolia/cache", {
"headers": {
"x-api-key": "2Zg8!d2WAHIUW?pCO28cVjfOt9seOWPx@2j"
},
"body": JSON.stringify({
url: "https://6UJ1I5A072-2.algolianet.com/1/indexes/assets/query?x-algolia-application-id=6UJ1I5A072&x-algolia-api-key=e93907f4f65fb1d9f813957bdc344892",
params: new URLSearchParams({ ...defaultParams, ...params }).toString()
}),
"method": "POST",
})
return await response.json()
}
const callAcl = async ({ id, name }) => {
const response = await fetch("https://quixel.com/v1/acl", {
"headers": {
"authorization": "Bearer " + authToken,
"content-type": "application/json;charset=UTF-8",
},
"body": JSON.stringify({ assetID: id }),
"method": "POST",
});
const json = await response.json()
if (json?.isError) {
console.error(` --> **UNABLE TO ADD ITEM** Item ${id} | ${name} (${json?.msg})`)
} else {
console.log(` --> ADDED ITEM Item ${id} | ${name}`)
}
}
const callAcquired = async () => {
const response = await fetch("https://quixel.com/v1/assets/acquired", {
"headers": {
"authorization": "Bearer " + authToken,
"content-type": "application/json;charset=UTF-8",
},
"method": "GET",
});
return await response.json()
}
// 1. Check token exist, quixel API needs it
console.log("-> Checking Auth API Token...")
let authToken = ""
try {
const authCookie = getCookie("auth") ?? "{}"
authToken = JSON.parse(decodeURIComponent(authCookie))?.token
if (!authToken) {
return console.error("-> Error: cannot find authentication token. Please login again.")
}
} catch (_) {
return console.error("-> Error: cannot find authentication token. Please login again.")
}
// 2. Get all currently acquired items
console.log("-> Get Acquired Items...")
const acquiredItems = (await callAcquired()).map(a => a.assetID)
// 3. Get total count of items
console.log("-> Getting Total Number of Pages....")
const { nbPages: totalPages, hitsPerPage: itemsPerPage, nbHits: totalItems } = await callCacheApi()
console.log("-> ==============================================")
console.log(`-> Total # of items: ${totalItems}`)
console.log(`-> ${totalPages} total pages with ${itemsPerPage} per page`)
console.log(`-> Total Items to add: ${(totalItems - acquiredItems.length)}.`)
console.log("-> ==============================================")
if (!confirm(`Click OK to start adding ${(totalItems - acquiredItems.length)} items in your account.`)) return
// Loop
for (let pageIdx = startPage || 0; pageIdx < totalPages; pageIdx++) {
console.log(`-> ======================= PAGE ${pageIdx + 1}/${totalPages} START =======================`)
console.log("-> Getting Items from page " + (pageIdx + 1) + " ...")
const { hits: items } = await callCacheApi({ page: pageIdx })
console.log("-> Adding non-acquired items...")
// Filter out owned items
const unownedItems = items.filter(i => !acquiredItems.includes(i.id))
const aclPromises = unownedItems.map(callAcl)
await Promise.all(aclPromises)
console.log(`-> ======================= PAGE ${pageIdx + 1}/${totalPages} COMPLETED =======================`)
if (autoClearConsole) console.clear() // Fix the issue that too much log hangs the console. Set autoClearConsole = false to keep the logs
}
console.log("-> Getting new acquired info...")
// Get acquired items again
const newItemsAcquired = (await callAcquired()).length
const newTotalCount = (await callCacheApi()).nbHits
console.log(`-> Completed. Your account now have a total of ${newItemsAcquired} out of ${newTotalCount} items.`)
alert(`-> Your account now have a total of ${newItemsAcquired} out of ${newTotalCount} items.\n\nIf you find some items missing, try refresh the page and run the script again.`)
})())
@willemgrooters
Copy link

willemgrooters commented Sep 19, 2024 via email

@Kaijiroo
Copy link

Thanks <3

@Igor-Vuk
Copy link

will this megascans site continue to operate or we need to download those assets to local drive in order to access them in future?

@Matthewruth71
Copy link

will this megascans site continue to operate or we need to download those assets to local drive in order to access them in future?

As far as i understood, all the assets library will transfer from Quixel to Fab , if you used your epic account to claim them.
Either way , you can just reclaim the all the assets for free on FAB until the end of 2024 , so don't worry.
So you won't need to to download , it would require at least a 30TB SSD.

@Igor-Vuk
Copy link

will this megascans site continue to operate or we need to download those assets to local drive in order to access them in future?

As far as i understood, all the assets library will transfer from Quixel to Fab , if you used your epic account to claim them. Either way , you can just reclaim the all the assets for free on FAB until the end of 2024 , so don't worry. So you won't need to to download , it would require at least a 30TB SSD.

I see. So if we run this script on megascan and got all the assets, they are going to get transferred to Fab when site launches? No need to claim them there again. In case we need to claim them again, we will be able to until the end of the year because we did it on megascan now.

@Matthewruth71
Copy link

Pretty much

@rinadyx
Copy link

rinadyx commented Sep 19, 2024

THANK YOU

@HuaChayu
Copy link

I tried a few more times。。。purchased:19814 ....>18876
23333

@mai1015
Copy link

mai1015 commented Sep 19, 2024

THANK YOU!

@MasterBlasterX1X
Copy link

willemgrooters

Ah, ok. Thank you, @willemgrooters! That makes sense.

So once we get over the 18,876 number, we're good to go in theory, correct?
We don't have to re-run the script and "add -117"? Lol.
I was afraid the off numbers meant the script might have accidentally missed some assets, and they wouldn't transfer over when everything switches to Fab.com

In any event, I plan on double-checking, and perhaps rerunning the script once everything transfers over, and again one more time at 10pm on New Year's Eve to capture any new stuff that gets released between now and the end of 2024.
Do you think we will need a new script when things transfer over to Fab.com, or can we re-use this one if we need to?

Thank you very much again for helping clear things up! :-) 👍

@Igor-Vuk
Copy link

Igor-Vuk commented Sep 19, 2024 •
edited
Loading

I also run a script few times and at the end I ended up with 19794 out of 18876. I run the script again, it said its gona remove the exces files but it just went through pages and I am left with19891 still. If someone knows how to sync the numbers please let me know.

@stl3
Copy link

stl3 commented Sep 19, 2024

Yeah I ran the primary script on the page and it hung twice so I tried vanthunder's version and it eventually completed but now it's showing a higher number, 19,513. Script shows there are 18,874 total items, the page itself is now showing 18,876 and that I have -639 to add.

image
image
image

I'm just concerned about that excess 19,513 number, like others have been getting (though not the exact same value).

@zheng95z
Copy link

Thanks for the amazing script!

I'm also wondering if there is way to download them using the API.

@ewebgh33
Copy link

Lol yeah got 19,000something scans...

@Sergeit
Copy link

Sergeit commented Sep 20, 2024

is there the same script so that you can pick up free libraries in epic games for the unreal engine?

@shadow211
Copy link

Many thanks! That helps a lot!

@Patchmonk
Copy link

Thank you so much for the script. you are awesome!!!

@BlackAngelYPK
Copy link

BlackAngelYPK commented Sep 20, 2024 •
edited
Loading

I made a few additions to the code to limit errors and check for duplicates for those who had already acquired certain assets.
I tested on Chrome and the script seems more robust and the results more consistent.

I'll share it with you here :

(async (startPage = 0, autoClearConsole = true) => {
  if (autoClearConsole) console.log("Note: Console clearing is disabled due to 'Preserve log' option.");

  // Constants for request management
  const MAX_CONCURRENT_REQUESTS = 5;
  const MAX_CONCURRENT_BATCHES = 5;

  // Define global variables
  let totalPages = 0;
  let acquiredItems = [];
  let authToken = '';

  // Utility functions
  const getCookie = (name) => {
    const match = document.cookie.match(new RegExp(`(^| )${name}=([^;]+)`));
    return match ? decodeURIComponent(match[2]) : null;
  };

  const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

  const fetchWithTimeout = async (url, options = {}) => {
    const { timeout = 10000 } = options;
    return Promise.race([
      fetch(url, options),
      new Promise((_, reject) => setTimeout(() => reject(new Error('Request timeout')), timeout))
    ]);
  };

  const handleJsonResponse = async (response) => {
    if (!response.ok) {
      throw new Error(`Error ${response.status}: ${response.statusText}`);
    }
    const contentType = response.headers.get("content-type");
    if (contentType && contentType.includes("application/json")) {
      return response.json();
    } else {
      const text = await response.text();
      throw new Error(`Expected JSON but received ${contentType}: ${text}`);
    }
  };

  const retryOperation = async (operation, delay, retries) => {
    let lastError;
    for (let attempt = 1; attempt <= retries; attempt++) {
      try {
        return await operation();
      } catch (error) {
        lastError = error;
        console.warn(`Attempt ${attempt} failed: ${error.message}. Retrying in ${delay}ms...`);
        await sleep(delay);
        delay *= 2;
      }
    }
    throw lastError;
  };

  const checkDataIntegrity = (items) => {
    const seenIdentifiers = new Set();
    const duplicates = new Set();
    items.forEach(item => {
      if (seenIdentifiers.has(item)) {
        duplicates.add(item);
      }
      seenIdentifiers.add(item);
    });
    if (duplicates.size) {
      console.warn("Duplicates found in items:", Array.from(duplicates));
    }
    return Array.from(seenIdentifiers).filter(item => !duplicates.has(item));
  };

  // Define the initialize function
  const initialize = async () => {
    console.log("Initializing...");
    try {
      const authCookie = getCookie("auth") ?? "{}";
      authToken = JSON.parse(decodeURIComponent(authCookie))?.token;
      if (!authToken) {
        throw new Error("Authentication token not found. Please log in again.");
      }

      console.log("Fetching acquired items...");
      acquiredItems = (await callAcquired()).map(a => a.assetID);

      console.log("Fetching total number of pages...");
      const initialData = await callCacheApi();
      totalPages = initialData.nbPages;
      itemsPerPage = initialData.hitsPerPage;
      totalItems = initialData.nbHits;

      console.log("Initialization complete.");
    } catch (error) {
      console.error("Error during initialization:", error.message);
      throw error; // Ensure the error propagates if initialization fails
    }
  };

  // Call Cache API
  const callCacheApi = async (params = {}) => {
    const defaultParams = {
      page: 0,
      maxValuesPerFacet: 1000,
      hitsPerPage: 1000,
      attributesToRetrieve: "id,name"
    };

    const fetchData = async () => {
      const response = await fetchWithTimeout("https://proxy-algolia-prod.quixel.com/algolia/cache", {
        headers: {
          "x-api-key": "2Zg8!d2WAHIUW?pCO28cVjfOt9seOWPx@2j",
        },
        body: JSON.stringify({
          url: "https://6UJ1I5A072-2.algolianet.com/1/indexes/assets/query?x-algolia-application-id=6UJ1I5A072&x-algolia-api-key=e93907f4f65fb1d9f813957bdc344892",
          params: new URLSearchParams({ ...defaultParams, ...params }).toString(),
        }),
        method: "POST",
      });
      return await handleJsonResponse(response);
    };

    return await retryOperation(fetchData, 2000, 5);
  };

  // Check if item exists
  const checkIfItemExists = async (id) => {
    try {
      const response = await fetchWithTimeout(`https://quixel.com/v1/assets/${id}`, {
        headers: {
          authorization: `Bearer ${authToken}`,
          "content-type": "application/json;charset=UTF-8",
        },
        method: "GET",
      });

      if (!response.ok) {
        if (response.status === 404) {
          return false;
        } else {
          throw new Error(`Error ${response.status}: ${response.statusText}`);
        }
      }

      return true;
    } catch (error) {
      console.error(`Failed to check if item exists: ${error.message}`);
      throw error;
    }
  };

  // Call ACL API
  const callAcl = async ({ id, name }) => {
    const fetchData = async () => {
      if (await checkIfItemExists(id)) {
        console.log(`Item ${name} (ID: ${id}) already exists. Skipping.`);
        return;
      }

      const response = await fetchWithTimeout("https://quixel.com/v1/acl", {
        headers: {
          authorization: `Bearer ${authToken}`,
          "content-type": "application/json;charset=UTF-8",
        },
        body: JSON.stringify({ assetID: id }),
        method: "POST",
      });

      const json = await handleJsonResponse(response);
      if (json?.isError) throw new Error(`Failed to add item ${id} | ${name}: ${json?.msg}`);
      console.log(`Added item ${id} | ${name}`);
    };

    return await retryOperation(fetchData, 2000, 5);
  };

  // Call Acquired API
  const callAcquired = async () => {
    const fetchData = async () => {
      const response = await fetchWithTimeout("https://quixel.com/v1/assets/acquired", {
        headers: {
          authorization: `Bearer ${authToken}`,
          "content-type": "application/json;charset=UTF-8",
        },
        method: "GET",
      });
      return await handleJsonResponse(response);
    };

    return await retryOperation(fetchData, 2000, 5);
  };

  // Identify unacquired items
  const identifyUnacquiredItems = async () => {
    const unacquiredItems = [];
    const batchSize = Math.ceil(totalPages / MAX_CONCURRENT_BATCHES);

    for (let batch = 0; batch < MAX_CONCURRENT_BATCHES; batch++) {
      const start = batch * batchSize;
      const end = Math.min((batch + 1) * batchSize, totalPages);
      console.log(`Fetching items from pages ${start + 1} to ${end}...`);

      const batchPromises = [];
      for (let pageIdx = start; pageIdx < end; pageIdx++) {
        batchPromises.push((async () => {
          try {
            const pageData = await callCacheApi({ page: pageIdx });
            const pageItems = pageData.hits;
            const pageUnacquiredItems = pageItems.filter(i => !acquiredItems.includes(i.id));
            unacquiredItems.push(...pageUnacquiredItems);
            await sleep(2000); // Delay between page fetches
          } catch (error) {
            console.error(`Error fetching page ${pageIdx}: ${error.message}`);
          }
        })());
      }
      await Promise.all(batchPromises);

      if (batch < MAX_CONCURRENT_BATCHES - 1) {
        console.log(`Waiting before next batch...`);
        await sleep(10000); // Delay between batches
      }
    }

    return unacquiredItems;
  };

  // Process unacquired items
  const processUnacquiredItems = async (items) => {
    const queue = [...items];
    const processedItems = new Set();

    const processQueue = async () => {
      const workers = [];
      while (queue.length) {
        const item = queue.shift();
        if (!processedItems.has(item.id)) {
          workers.push((async () => {
            try {
              await callAcl(item);
              processedItems.add(item.id);
            } catch (error) {
              console.error(`Failed to add item ${item.id}: ${error.message}`);
              localStorage.setItem(`failedItem_${item.id}`, JSON.stringify({ item, error: error.message }));
            }
          })());
        }
        if (workers.length >= MAX_CONCURRENT_REQUESTS) {
          await Promise.all(workers);
          workers.length = 0;
        }
      }
      await Promise.all(workers);
    };

    await processQueue();
  };

  // Finalize process
  const finalize = async () => {
    console.log("-> Fetching new acquisition info...");
    try {
      const newAcquiredItems = await callAcquired();
      const newItemsAcquired = newAcquiredItems.length;
      const newTotalCount = (await callCacheApi()).nbHits;

      console.log(`-> Completed. Your account now has a total of ${newItemsAcquired} out of ${newTotalCount} items.`);
      alert(`-> Your account now has a total of ${newItemsAcquired} out of ${newTotalCount} items.\n\nIf you find some items missing, try refreshing the page and running the script again.`);
    } catch (error) {
      console.error(`Error during finalization: ${error.message}`);
    }
  };

  // Main execution
  try {
    const savedPage = localStorage.getItem('currentPage');
    if (savedPage !== null) {
      startPage = parseInt(savedPage, 10);
      console.log(`-> Resuming from page ${startPage + 1}`);
    }

    await initialize();
    const unacquiredItems = await identifyUnacquiredItems();
    await processUnacquiredItems(unacquiredItems);
    await finalize();
    localStorage.removeItem('currentPage');
  } catch (error) {
    console.error(error.message);
    console.log("-> The script could not be completed.");
  }
})();

@mechloz
Copy link

mechloz commented Sep 20, 2024

Thank you!!

@bob4081
Copy link

bob4081 commented Sep 20, 2024

Legend! Thank you

@fhnb16
Copy link

fhnb16 commented Sep 20, 2024 •
edited
Loading

@BlackAngelYPK your message is broken because of the quotes ` that markdown threats as line of code, check preview of the message before sending.
Upd. Fixed

@Dylan-Komyuter
Copy link

@fhnb16 Thank you,you fixed the problem!

@ascent-sys
Copy link

Thank you!

@alsi1976
Copy link

Hi, thanks for this nice gift, I see that I can't past the code, and I read a Warnig any time I try can you help me?

"Warning: Don’t paste code into the DevTools Console that you don’t understand or haven’t reviewed yourself. This could allow attackers to steal your identity or take control of your computer. Please type ‘allow pasting’ below and hit Enter to allow pasting."

@clicexdice
Copy link

THANK YOU!!!

@Dylan-Komyuter
Copy link

Type ' allow pasting ' on the console,and then it will work.
Console is where you copy the code.
Snipaste_2024-09-20_18-40-12

@tongmon
Copy link

tongmon commented Sep 20, 2024

Hi, thanks for this nice gift, I see that I can't past the code, and I read a Warnig any time I try can you help me?

"Warning: Don’t paste code into the DevTools Console that you don’t understand or haven’t reviewed yourself. This could allow attackers to steal your identity or take control of your computer. Please type ‘allow pasting’ below and hit Enter to allow pasting."

Just type the "allow pasting" text into ur console and u r ready to go.

@MiguelAlejandria
Copy link

would not making a large number of API calls trigger rate limits or even get your account flagged for abuse by the site?

@BlackAngelYPK
Copy link

BlackAngelYPK commented Sep 20, 2024 •
edited
Loading

@fhnb16 sorry, the display problem has been fixed in my previous post

@madox21888
Copy link

VM110:154 Item Desert Western Cliff Sheer XL 09 (ID: xgjodgt) already exists. Skipping. guys i always face this problem is there any solution ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment