User:Robbie/linux wget: Difference between revisions

(One intermediate revision by the same user not shown)

Line 29:

1. **Create a Text File with URLs:**

- Create a text file (e.g., `urls.txt`) and list all the URLs of the directories you want to download, one per line.

Example `urls.txt`:

```

http://example.com/directory1/

http://example.com/directory2/

http://example.com/directory3/

...

```

2. **Use wget with the -i Option:**

- Use the following command to download all the directories listed in your `urls.txt` file:

```bash

wget -r -np -nH --cut-dirs=1 -i urls.txt

```

Explanation of the options:

- `-r`: Recursively download the directories.

- `-np`: No parent. This prevents `wget` from downloading parent directories.

- `-nH`: No host directories. This prevents `wget` from creating a directory named after the host.

- `--cut-dirs=1`: This option removes the specified number of directory components from the path. Adjust the number according to your directory structure.

- `-i urls.txt`: Specifies the input file containing the list of URLs.

3. **Adjust Options as Needed:**

- Depending on your server setup and directory structure, you might need to adjust the `--cut-dirs` option or add authentication options if required.

This approach allows you to efficiently download multiple directories using `wget` by leveraging a list of URLs.

Line 70:

```bash

while read dir; do

wget -r -np -nH --cut-dirs=1 "http://base-url.com/$dir"

done < list.txt

```

@@ Line 29: / Line 29: @@
 . **Create a Text File with URLs:**
-   - Create a text file (e.g., `urls.txt`) and list all the URLs of the directories you want to download, one per line.
+- Create a text file (e.g., `urls.txt`) and list all the URLs of the directories you want to download, one per line.
-   Example `urls.txt`:
+Example `urls.txt`:
-   ```
+```
-   http://example.com/directory1/
+http://example.com/directory1/
-   http://example.com/directory2/
+http://example.com/directory2/
-   http://example.com/directory3/
+http://example.com/directory3/
-   ...
+...
-   ```
+```
 . **Use wget with the -i Option:**
-   - Use the following command to download all the directories listed in your `urls.txt` file:
+- Use the following command to download all the directories listed in your `urls.txt` file:
-   ```bash
+```bash
-   wget -r -np -nH --cut-dirs=1 -i urls.txt
+wget -r -np -nH --cut-dirs=1 -i urls.txt
-   ```
+```
-   Explanation of the options:
+Explanation of the options:
-   - `-r`: Recursively download the directories.
+- `-r`: Recursively download the directories.
-   - `-np`: No parent. This prevents `wget` from downloading parent directories.
+- `-np`: No parent. This prevents `wget` from downloading parent directories.
-   - `-nH`: No host directories. This prevents `wget` from creating a directory named after the host.
+- `-nH`: No host directories. This prevents `wget` from creating a directory named after the host.
-   - `--cut-dirs=1`: This option removes the specified number of directory components from the path. Adjust the number according to your directory structure.
+- `--cut-dirs=1`: This option removes the specified number of directory components from the path. Adjust the number according to your directory structure.
-   - `-i urls.txt`: Specifies the input file containing the list of URLs.
+- `-i urls.txt`: Specifies the input file containing the list of URLs.
 . **Adjust Options as Needed:**
-   - Depending on your server setup and directory structure, you might need to adjust the `--cut-dirs` option or add authentication options if required.
+- Depending on your server setup and directory structure, you might need to adjust the `--cut-dirs` option or add authentication options if required.
 This approach allows you to efficiently download multiple directories using `wget` by leveraging a list of URLs.
@@ Line 70: / Line 70: @@
 ```bash
 while read dir; do
-    wget -r -np -nH --cut-dirs=1 "http://base-url.com/$dir"
+ wget -r -np -nH --cut-dirs=1 "http://base-url.com/$dir"
 done < list.txt
 ```