Added the ability to crawl content links

config.ts

@@ -12,7 +12,13 @@ type Config = {
   /** File name for the finished data */
   outputFileName: string;
   /** Optional cookie to be set. E.g. for Cookie Consent */
-  cookie?: {name: string; value: string}
+  cookie?: { name: string; value: string };
+  /** Selector for the area of the page that should be processed. */
+  withoutSelector: string;
+  /** List of HTML attributes to retain during content processing. */
+  attributeWhitelist: string[];
+  /** Flag to determine if content links should be crawled. */
+  isContentLink: boolean;
   /** Optional function to run for each page found */
   onVisitPage?: (options: {
     page: Page;
@@ -21,9 +27,12 @@ type Config = {
 };
 
 export const config: Config = {
-  url: "https://www.builder.io/c/docs/developers",
-  match: "https://www.builder.io/c/docs/**",
-  selector: `.docs-builder-container`,
+  url: "https://react.dev/learn",
+  match: "https://react.dev/learn",
+  selector: `main`,
+  isContentLink: false,
+  withoutSelector: `main`,
+  attributeWhitelist: ["href", "title"],
   maxPagesToCrawl: 50,
-  outputFileName: "output.json",
+  outputFileName: "nuxt.json",
 };

nuxt.json

@@ -0,0 +1,7 @@
+[
+  {
+    "title": "Quick Start – React",
+    "url": "https://react.dev/learn",
+    "html": "LEARN REACT\nQuick Start\n\nWelcome to the React documentation! This page will give you an introduction to the 80% of React concepts that you will use on a daily basis.\n\nYou will learn\nHow to create and nest components\nHow to add markup and styles\nHow to display data\nHow to render conditions and lists\nHow to respond to events and update the screen\nHow to share data between components\nCreating and nesting components \n\nReact apps are made out of components. A component is a piece of the UI (user interface) that has its own logic and appearance. A component can be as small as a button, or as large as an entire page.\n\nReact components are JavaScript functions that return markup:\n\nfunction MyButton() {\n\n  return (\n\n    <button>I'm a button</button>\n\n  );\n\n}\n\nNow that you’ve declared MyButton, you can nest it into another component:\n\nexport default function MyApp() {\n\n  return (\n\n    <div>\n\n      <h1>Welcome to my app</h1>\n\n      <MyButton />\n\n    </div>\n\n  );\n\n}\n\nNotice that <MyButton /> starts with a capital letter. That’s how you know it’s a React component. React component names must always start with a capital letter, while HTML tags must be lowercase.\n\nHave a look at the result:\n\nApp.js\nDownload\nReset\nFork\nfunction MyButton() {\n  return (\n    <button>\n      I'm a button\n    </button>\n  );\n}\n\nexport default function MyApp() {\n  return (\n    <div>\n      <h1>Welcome to my app</h1>\n      <MyButton />\n    </div>\n  );\n}\n\n\nShow more\n\nThe export default keywords specify the main component in the file. If you’re not familiar with some piece of JavaScript syntax, MDN and javascript.info have great references.\n\nWriting markup with JSX \n\nThe markup syntax you’ve seen above is called JSX. It is optional, but most React projects use JSX for its convenience. All of the tools we recommend for local development support JSX out of the box.\n\nJSX is stricter than HTML. You have to close tags like <br />. Your component also can’t return multiple JSX tags. You have to wrap them into a shared parent, like a <div>...</div> or an empty <>...</> wrapper:\n\nfunction AboutPage() {\n\n  return (\n\n    <>\n\n      <h1>About</h1>\n\n      <p>Hello there.<br />How do you do?</p>\n\n    </>\n\n  );\n\n}\n\nIf you have a lot of HTML to port to JSX, you can use an online converter.\n\nAdding styles \n\nIn React, you specify a CSS class with className. It works the same way as the HTML class attribute:\n\n<img className=\"avatar\" />\n\nThen you write the CSS rules for it in a separate CSS file:\n\n/* In your CSS */\n\n.avatar {\n\n  border-radius: 50%;\n\n}\n\nReact does not prescribe how you add CSS files. In the simplest case, you’ll add a <link> tag to your HTML. If you use a build tool or a framework, consult its documentation to learn how to add a CSS file to your project.\n\nDisplaying data \n\nJSX lets you put markup into JavaScript. Curly braces let you “escape back” into JavaScript so that you can embed some variable from your code and display it to the user. For example, this will display user.name:\n\nreturn (\n\n  <h1>\n\n    {user.name}\n\n  </h1>\n\n);\n\nYou can also “escape into JavaScript” from JSX attributes, but you have to use curly braces instead of quotes. For example, className=\"avatar\" passes the \"avatar\" string as the CSS class, but src={user.imageUrl} reads the JavaScript user.imageUrl variable value, and then passes that value as the src attribute:\n\nreturn (\n\n  <img\n\n    className=\"avatar\"\n\n    src={user.imageUrl}\n\n  />\n\n);\n\nYou can put more complex expressions inside the JSX curly braces too, for example, string concatenation:\n\nApp.js\nDownload\nReset\nFork\nconst user = {\n  name: 'Hedy Lamarr',\n  imageUrl: 'https://i.imgur.com/yXOvdOSs.jpg',\n  imageSize: 90,\n};\n\nexport default function Profile() {\n  return (\n    <>\n      <h1>{user.name}</h1>\n      <img\n        className=\"avatar\"\n        src={user.imageUrl}\n        alt={'Photo of ' + user.name}\n        style={{\n          width: user.imageSize,\n          height: user.imageSize\n        }}\n      />\n    </>\n  );\n}\n\n\nShow more\n\nIn the above example, style={{}} is not a special syntax, but a regular {} object inside the style={ } JSX curly braces. You can use the style attribute when your styles depend on JavaScript variables.\n\nConditional rendering \n\nIn React, there is no special syntax for writing conditions. Instead, you’ll use the same techniques as you use when writing regular JavaScript code. For example, you can use an if statement to conditionally include JSX:\n\nlet content;\n\nif (isLoggedIn) {\n\n  content = <AdminPanel />;\n\n} else {\n\n  content = <LoginForm />;\n\n}\n\nreturn (\n\n  <div>\n\n    {content}\n\n  </div>\n\n);\n\nIf you prefer more compact code, you can use the conditional ? operator. Unlike if, it works inside JSX:\n\n<div>\n\n  {isLoggedIn ? (\n\n    <AdminPanel />\n\n  ) : (\n\n    <LoginForm />\n\n  )}\n\n</div>\n\nWhen you don’t need the else branch, you can also use a shorter logical && syntax:\n\n<div>\n\n  {isLoggedIn && <AdminPanel />}\n\n</div>\n\nAll of these approaches also work for conditionally specifying attributes. If you’re unfamiliar with some of this JavaScript syntax, you can start by always using if...else.\n\nRendering lists \n\nYou will rely on JavaScript features like for loop and the array map() function to render lists of components.\n\nFor example, let’s say you have an array of products:\n\nconst products = [\n\n  { title: 'Cabbage', id: 1 },\n\n  { title: 'Garlic', id: 2 },\n\n  { title: 'Apple', id: 3 },\n\n];\n\nInside your component, use the map() function to transform an array of products into an array of <li> items:\n\nconst listItems = products.map(product =>\n\n  <li key={product.id}>\n\n    {product.title}\n\n  </li>\n\n);\n\n\n\nreturn (\n\n  <ul>{listItems}</ul>\n\n);\n\nNotice how <li> has a key attribute. For each item in a list, you should pass a string or a number that uniquely identifies that item among its siblings. Usually, a key should be coming from your data, such as a database ID. React uses your keys to know what happened if you later insert, delete, or reorder the items.\n\nApp.js\nDownload\nReset\nFork\nconst products = [\n  { title: 'Cabbage', isFruit: false, id: 1 },\n  { title: 'Garlic', isFruit: false, id: 2 },\n  { title: 'Apple', isFruit: true, id: 3 },\n];\n\nexport default function ShoppingList() {\n  const listItems = products.map(product =>\n    <li\n      key={product.id}\n      style={{\n        color: product.isFruit ? 'magenta' : 'darkgreen'\n      }}\n    >\n      {product.title}\n    </li>\n  );\n\n  return (\n    <ul>{listItems}</ul>\n  );\n}\n\n\nShow more\nResponding to events \n\nYou can respond to events by declaring event handler functions inside your components:\n\nfunction MyButton() {\n\n  function handleClick() {\n\n    alert('You clicked me!');\n\n  }\n\n\n\n  return (\n\n    <button onClick={handleClick}>\n\n      Click me\n\n    </button>\n\n  );\n\n}\n\nNotice how onClick={handleClick} has no parentheses at the end! Do not call the event handler function: you only need to pass it down. React will call your event handler when the user clicks the button.\n\nUpdating the screen \n\nOften, you’ll want your component to “remember” some information and display it. For example, maybe you want to count the number of times a button is clicked. To do this, add state to your component.\n\nFirst, import useState from React:\n\nimport { useState } from 'react';\n\nNow you can declare a state variable inside your component:\n\nfunction MyButton() {\n\n  const [count, setCount] = useState(0);\n\n  // ...\n\nYou’ll get two things from useState: the current state (count), and the function that lets you update it (setCount). You can give them any names, but the convention is to write [something, setSomething].\n\nThe first time the button is displayed, count will be 0 because you passed 0 to useState(). When you want to change state, call setCount() and pass the new value to it. Clicking this button will increment the counter:\n\nfunction MyButton() {\n\n  const [count, setCount] = useState(0);\n\n\n\n  function handleClick() {\n\n    setCount(count + 1);\n\n  }\n\n\n\n  return (\n\n    <button onClick={handleClick}>\n\n      Clicked {count} times\n\n    </button>\n\n  );\n\n}\n\nReact will call your component function again. This time, count will be 1. Then it will be 2. And so on.\n\nIf you render the same component multiple times, each will get its own state. Click each button separately:\n\nApp.js\nDownload\nReset\nFork\nimport { useState } from 'react';\n\nexport default function MyApp() {\n  return (\n    <div>\n      <h1>Counters that update separately</h1>\n      <MyButton />\n      <MyButton />\n    </div>\n  );\n}\n\nfunction MyButton() {\n  const [count, setCount] = useState(0);\n\n  function handleClick() {\n    setCount(count + 1);\n  }\n\n  return (\n    <button onClick={handleClick}>\n      Clicked {count} times\n    </button>\n  );\n}\n\n\nShow more\n\nNotice how each button “remembers” its own count state and doesn’t affect other buttons.\n\nUsing Hooks \n\nFunctions starting with use are called Hooks. useState is a built-in Hook provided by React. You can find other built-in Hooks in the API reference. You can also write your own Hooks by combining the existing ones.\n\nHooks are more restrictive than other functions. You can only call Hooks at the top of your components (or other Hooks). If you want to use useState in a condition or a loop, extract a new component and put it there.\n\nSharing data between components \n\nIn the previous example, each MyButton had its own independent count, and when each button was clicked, only the count for the button clicked changed:\n\nInitially, each MyButton’s count state is 0\n\nThe first MyButton updates its count to 1\n\nHowever, often you’ll need components to share data and always update together.\n\nTo make both MyButton components display the same count and update together, you need to move the state from the individual buttons “upwards” to the closest component containing all of them.\n\nIn this example, it is MyApp:\n\nInitially, MyApp’s count state is 0 and is passed down to both children\n\nOn click, MyApp updates its count state to 1 and passes it down to both children\n\nNow when you click either button, the count in MyApp will change, which will change both of the counts in MyButton. Here’s how you can express this in code.\n\nFirst, move the state up from MyButton into MyApp:\n\nexport default function MyApp() {\n\n  const [count, setCount] = useState(0);\n\n\n\n  function handleClick() {\n\n    setCount(count + 1);\n\n  }\n\n\n\n  return (\n\n    <div>\n\n      <h1>Counters that update separately</h1>\n\n      <MyButton />\n\n      <MyButton />\n\n    </div>\n\n  );\n\n}\n\n\n\nfunction MyButton() {\n\n  // ... we're moving code from here ...\n\n}\n\nThen, pass the state down from MyApp to each MyButton, together with the shared click handler. You can pass information to MyButton using the JSX curly braces, just like you previously did with built-in tags like <img>:\n\nexport default function MyApp() {\n\n  const [count, setCount] = useState(0);\n\n\n\n  function handleClick() {\n\n    setCount(count + 1);\n\n  }\n\n\n\n  return (\n\n    <div>\n\n      <h1>Counters that update together</h1>\n\n      <MyButton count={count} onClick={handleClick} />\n\n      <MyButton count={count} onClick={handleClick} />\n\n    </div>\n\n  );\n\n}\n\nThe information you pass down like this is called props. Now the MyApp component contains the count state and the handleClick event handler, and passes both of them down as props to each of the buttons.\n\nFinally, change MyButton to read the props you have passed from its parent component:\n\nfunction MyButton({ count, onClick }) {\n\n  return (\n\n    <button onClick={onClick}>\n\n      Clicked {count} times\n\n    </button>\n\n  );\n\n}\n\nWhen you click the button, the onClick handler fires. Each button’s onClick prop was set to the handleClick function inside MyApp, so the code inside of it runs. That code calls setCount(count + 1), incrementing the count state variable. The new count value is passed as a prop to each button, so they all show the new value. This is called “lifting state up”. By moving state up, you’ve shared it between components.\n\nApp.js\nDownload\nReset\nFork\nimport { useState } from 'react';\n\nexport default function MyApp() {\n  const [count, setCount] = useState(0);\n\n  function handleClick() {\n    setCount(count + 1);\n  }\n\n  return (\n    <div>\n      <h1>Counters that update together</h1>\n      <MyButton count={count} onClick={handleClick} />\n      <MyButton count={count} onClick={handleClick} />\n    </div>\n  );\n}\n\nfunction MyButton({ count, onClick }) {\n  return (\n    <button onClick={onClick}>\n      Clicked {count} times\n    </button>\n  );\n}\n\n\nShow more\nNext Steps \n\nBy now, you know the basics of how to write React code!\n\nCheck out the Tutorial to put them into practice and build your first mini-app with React.\n\nNEXT\nTutorial: Tic-Tac-Toe\n\nHow do you like these docs?\n\nTake our survey!\n©2023\nLearn React\nQuick Start\nInstallation\nDescribing the UI\nAdding Interactivity\nManaging State\nEscape Hatches\nAPI Reference\nReact APIs\nReact DOM APIs\nCommunity\nCode of Conduct\nMeet the Team\nDocs Contributors\nAcknowledgements\nMore\nBlog\nReact Native\nPrivacy\nTerms"
+  }
+]