Sed help needed!
I asked ChatGPT to help me create a function to help me parse YT URL's. FYI a YT URL consists of a string where it can have the video id alone, or the video id plus extra metadata. The video id is 11 characters long. Anything after it, after a "&" or "?" character is just extra metadata These 3 sample YT URL's are the exact same video: youtu.be/watch?=12345678901 and `youtu.be/watch?=12345678901&pp=abcdefghij%3D%3D` and `youtu.be/watch?=1234678901?si=abcd1234`
The code that ChatGPT gave me was: youtube_url=$1video_id=$(echo "$youtube_url" | sed -n -e 's/^.*[?&]v=\([^&]*\).*$/\1/p; s/^.*`youtu.be`\/\([^?]*\).*$/\1/p')
if [ -n "$video_id" ]; thenecho "$video_id"
else echo "Video ID not found in the given URL."
This works, basically. But my problem is that it only really works correctly with URLs of the first type. This will echo the correct video id,12345678901 and end. Which is the correct response I want with all 3 URLs. With the second or third type of URLs it will echo a Job id# then12345678901 and waits for me to hit ENTER before echoing a job id# with +Done and the function name. This is an unwanted result as I just want the result and no job id#s or having to press ENTER or an echo of the function name. What do I have to add for this to happen???
https://redd.it/18n7utu
@r_bash
Can I have a script output live/changing data?
Say I'm running a few dozen processes 3 at a time. I'd like to have the output show which is running, and update the existing output it to say when they're done. Is this possible?
eg:
Job 1: running
Job 2: running
Job 3: running
which would then change to:
Job 1: completed
Job 2: completed
Job 3: running
Job 4: running
Job 5: running
I'd prefer to do this without clearing the whole terminal.
https://redd.it/18mwl2z
@r_bash
How to handle empty values in bash
I'm trying to obtain the value of the "DEFAULT SOURCE VERSION" column for both rows. The issue arises from the fact that the "DEFAULT SOURCE VERSION" is located in the 4th column. However, there is an issue with the "lawn-mower" row where the column "PREVIOUS DEFAULT VERSION" is empty. As a result, when accessing the 4th column using a bash command, the correct value (1.25.0) is retrieved for "earth-shaker," but for "lawn-mower," it returns 1 due to the absence of a value in the "PREVIOUS DEFAULT VERSION" column. The expected output is "1.25.0" for both rows, but the actual output shows "1.25.0" for "earth-shaker" and 1 for "lawn-mower.
Tabular data
The columns here are not separated by any specific delimeter as versions may vary. I tried to use bash arrays to solve this issue but i was not able to resolve this issue .
​
OVERALL:
Expected output :
1.25.0
1.25.0
My Output:
1.25.0
1
​
Tabular data
NAME DEFAULT VERSION PREVIOUS DEFAULT VERSION DEFAULT SOURCE VERSION INSTALLED VERSION INSTALLED SOURCE VERSION UNKNOWN ERROR PENDING SUCCESS/DONE
earth-shaker 1 0 1.25.0 1 1.25.0 0 0 0 1
lawn-mower 1 1.25.0 1 1.25.0 0 0 0 1
https://redd.it/18m64pb
@r_bash
Tips for beginners
I’ve been thinking of starting learning bash scripting first quite a while and now that I have the time I want to know how to start. Can someone please suggest any links or materials?
https://redd.it/18lijx4
@r_bash
Why i can't get output when i use subshell?
I'm trying to solve OverTheWire's natas16 challenge its a simple command injection '$()' chars aren't restricted so i can run code through input. Tried something like "$(grep <token> /etc/natas_webpass/natas17)easters" as input it works but there is no output. When i read write-ups about natas16 similar ways are shared. If token in password file there is no output if not easters out. I can't understand why i can't get output when grep find something in password file?
lab is: http://natas16.natas.labs.overthewire.org/
username:natas16
password: TRD7iZrd5gATjj9PkPEuaOlfEjHqj32V
source code:
```<?
$key = "";
if(array_key_exists("needle", $_REQUEST)) {
$key = $_REQUEST["needle"\];
}
if($key != "") {
if(preg_match('/[;|&\\
'"\]/',$key)) {
print "Input contains an illegal character!";
} else {
passthru("grep -i "$key" dictionary.txt");
}
}
?>```
https://redd.it/18kdsef
@r_bash
Sometime "wc -l" is not enough to count all the outputs of a script, how can I solve?
I have this function to install programs using several scripts from a database:
function installarg(){
if grep -q 'read -r' ./$arg; then
$SUDOCOMMAND ./$arg | less -E
else
outputlines=$($SUDOCOMMAND ./$arg | wc -l)
for i in "$outputlines"; do
printf '\033['$i'A\033[K'
done
for i in "*"; do
echo $* 2>&1
done
fi
}
All this function should do is to replace extra outputs of the installation script.
During the installation, the progress bar of "wget" is shown, and each line is replaced depending on the state of the installation script.
Normally, on the same line, appear:
◆ RUNNING THE INSTALLATION SCRIPT FOR LXTASK
then something like this:
TaskManager-.0.1.9 100%===================> 2,97M 7,11MB/s in 0,4s
and finally
◆ "LXTASK" INSTALLED (4 MB OF DISK SPACE)
That said, this is what actually does:
-----------------------------------------------------------------------
>> START OF ALL INSTALLATION PROCESSES <<
-----------------------------------------------------------------------
◆ "FFWA-NETFLIX" INSTALLED (1 MB OF DISK SPACE)
index.html.tmp 100%===================> 2,94M 5,82MB/s in 0,5s
index.html.tmp 100%===================> 802 --.-KB/s in 0s
◆ "PALEMOON" INSTALLED (110 MB OF DISK SPACE)
◆ "FFWA-WHATSAPP" INSTALLED (1 MB OF DISK SPACE)
◆ "LXTASK" INSTALLED (4 MB OF DISK SPACE)
-----------------------------------------------------------------------
>> END OF ALL INSTALLATION PROCESSES <<
-----------------------------------------------------------------------
and this is what I'd like to see instead
-----------------------------------------------------------------------
>> START OF ALL INSTALLATION PROCESSES <<
-----------------------------------------------------------------------
◆ "FFWA-NETFLIX" INSTALLED (1 MB OF DISK SPACE)
◆ "PALEMOON" INSTALLED (110 MB OF DISK SPACE)
◆ "FFWA-WHATSAPP" INSTALLED (1 MB OF DISK SPACE)
◆ "LXTASK" INSTALLED (4 MB OF DISK SPACE)
-----------------------------------------------------------------------
>> END OF ALL INSTALLATION PROCESSES <<
-----------------------------------------------------------------------
The "wc -l" command is not enough for this, I'd like to list all the outputs of the script and grab them in the count to replace them (see that index.html.tmp 100%[===================>] and the doubled space between ffwa-whatsapp and lxtask).
How can I solve this?
PS: previously I had to use less -E, but now I'd like to show all the outputs without having to use less.
Please, help me.
https://redd.it/18jw9zi
@r_bash
Variable substitution in a command
So I'm making this bash script that essentially transfers files from a source directory to a destination directory and organises them based on their extensions. I've completed this part but now I want to add a flag to exclude certain extensions like -e. I have done this with getopts and it's working fine.
The problem I'm encounterung is while executing the find command that gets me the file paths. I'm building the conditional string based on the input to the -e flag.
The code for this part :
declare excluded_extensions="-name '*.*'"
if [ ! "$excluded_extensions" == "-name '*.*'" ]; then
extension_string="-not \("
for ext in $excluded_extensions; do
extension_string+=" -name '*.$ext' -o"
done
extension_string="${extension_string:0:-2}"
extension_string+="\)"
fi
The logic is that I set a default value to the variable which is `-name '*.*'`. So if the user doesn't want to exclude any extensions (so the -e is not used) the variable value is substituted as is : `-name '*.*'` which means find all files in the directory. But if there are any extensions specified by the user then it builds the string and it becomes `-not /( -name ext1 -o -name ext2 /)` and so on. Then the value is substituted in the find command:
`find source_dir -type f $extension_string`
This is to get all the file paths
I've echoed the content of the command with the string based on various inputs and the value is showing up properly formatted in the terminal.
However when I run it there's an error with find :
`find:paths must precede expression : ``\('``
I know the code and method is very messy so I would really appreciate any help or even if there's a better strategy to this. Researched a lot for this problem on stack overflow, chatgpt but no answer. Thanks in advance. Kindly let me know if there's anything more that I should explain about the script, I'll gladly do so.
https://redd.it/18j79wk
@r_bash
Double brackets quirk
Hi,
While I was messing with my code, I made what I thought was a typo but it seems to work anyway.
Notice how I'm not referencing the array item with ${codei}.
codes=(101 202 303)
i=0
echo $(( codesi == 101 ))
-> 1
echo $(( codesi == 202 ))
-> 0
What am I missing here? Is it a know behavior? I couldn't find anything in the references.
https://redd.it/18ixixg
@r_bash
Curl - Collection history fzf support
Hello
Are there any CLI utility that you came across where I can do the following using Curl
\- History of the past requests
\- Store it as a collection
\- Search using fzf like utility.
If none, anyone interested to build such a thing and I can initiate a thread on that.
​
https://redd.it/18it5fk
@r_bash
Why is this bash command not doing anything on macOS?
For some reason following command doesn't work on macOS in bash. It does work on other linux machine. Any idea why it doesn't work on macOS? ~ $ du -cBG . --max-depth=1 2>/dev/null | sort -n~ $ ~ $ which $SHELL/bin/bash
Just du works.
https://redd.it/18ihhfm
@r_bash
Resizing all images in a folder, with GUI progress bar, and total size
I created a simple script that uses imagemagick to go through all files in a folder and resize them, but I wanted to improve it, since the only way to monitor the progress of the command was to actually watch the files as they changed sizes (script is saved in *\~/.local/share/nautilus/scripts* so I can access it from a right-click menu).
I also have a habit of checking the folder size before and after, because I like seeing how much space it saved after the script has run. There are two scripts I use, one to resize to 4K (4000px on long edge), and one to resize to 1920 on the long edge.
I learned about *zenity* yesterday, and spent a few hours going through documentation and experimenting with the results. I don't have anyone else to share it with, so here it is!
#!/bin/bash
DIR="."
SIZE_ORIG=`du -h . | cut -f1`
# Starting Count
PROGRESS=0
NUMBER_OF_FILES=`find $DIR -maxdepth 1 -type f -name "*.*" | wc -l`
(for file in *.*; do
PROGRESS=$(( PROGRESS+1 ))
SIZE=`du -h . | cut -f1`
echo "# Resizing $file ... ( $SIZE_ORIG → $SIZE )"
mogrify -resize "1920x1920>" -quality 90% $file; sleep 0
echo $(( 100*$PROGRESS/$NUMBER_OF_FILES ))
done) | zenity --progress --width=420 --title "Resizing "$NUMBER_OF_FILES" files..." --percentage=0
The result looks like this:
​
https://preview.redd.it/mek0ykrts96c1.png?width=789&format=png&auto=webp&s=846aea13d344c3dee04e296c1e98e22c8671abfd
https://redd.it/18i9upl
@r_bash
Run a command as a non-root user when logged in as root
I have a script that requires root privileges and I don't want to hard code sudo (or doas) in the script. Thus, I run the script with sudo. So far, so simple.
However, some commands in the script have to be run as a non-root user. Is there a way to accomplish this?
https://redd.it/18i8a1r
@r_bash
Receive notification at SSH Login
Hi all!
I have a new machine on which I have set up an SSH server. I wanted to setup a little script so that I can receive a Telegram notification upon login. Following a few online guides I have put together the following script:
#!/usr/bin/env bash
TELEGRAMTOKEN="MYTOKEN"
CHATID="MYCHATID"
# Build the SSH login message
SSHMESSAGE="SSH login detected on $(hostname) at $(date) by user $(whoami) from $(echo $SSHCONNECTION | awk '{print $1}')"
# Send the message to Telegram
curl -s -X POST "https://api.telegram.org/bot$TELEGRAMTOKEN/sendMessage" \
-H 'Content-Type: application/json' \
-d '{"chatid": "'"$CHATID"'", "text": "'"$SSHMESSAGE"'"}' > /dev/null 2>&1
How can I have this running at each SSH login without interfering with the process? I had this set up in the past by modifying the SSH configuration under etc/ssh/sshd_config with
ForceCommand /usr/local/bin/sshloginnotify.sh; CANT REMEMBER
The command in the square bracket was added because initially the script would run at login, and log me out immediately after, but I can't remember how I had fixed that.
Ultimately however, completely removed the ForceCommand as it would not let me work remotely using Remote SSH login with Vscode.
Any help on how to set this up properly is highly appreciated, and sorry for my scarce knowledge, I am just getting into this!
https://redd.it/18hmv25
@r_bash
I want to "cat" some files with unknown names and a small pause in between
This can easly be explained with an example.
I have a directory with several txt files: 1.txt 2.txt 3.txt and I want to read 1.txt, then press enter and read 2.txt, press enter and read 3.txt. Instead of pressing enter I currently just use sleep 5, but I know how to change that later. However, the names are not 1 2 3 but something else I don't know, because I want to use this skript in several directories with different content.
​
Problem (or better said challenge, since there are no problems):
When I type cat *.txt it will display all .txt files, but I cannot read that fast. I would like to do something like cat 1.txt; sleep 5; clear; cat 2.txt; sleep 5; clear; cat 3.txt; just without typing every filename in there. Is there a way to read the contents of a directory and fill this out automatically?
https://redd.it/18hllm1
@r_bash
Techniques and methods for obtaining access to data protected by linux-based encryption – A reference guide for practitioners
https://www.sciencedirect.com/science/article/pii/S2666281723001816
https://redd.it/18h9atw
@r_bash
Some useful bash projects for linux servers as a helpdesk tier 2?
What I've made
- log vomitter (thing that vomits logs so that I can use logrotate and verify-I know that's stupid).
- glassfish/payara domain restart command.
That's all that I've needed and I've made them. Is there anything else that I can make? That will be useful to me a helpdesk tier 2?
https://redd.it/18mvtld
@r_bash
why is my script not working?
The script reads domain names from a text file, performs a DNS query using "dig +short NS" for each domain, and checks if "ns14.net" is present in the results. The issue is that, despite correct manual queries returning "ns14.net," the script consistently outputs "Nein" (No) for all domains, even if the correct answer would be Yes.
​
Can someone help?
#!/bin/bash
if $# -ne 1 ; then
echo "Verwendung: $0 inputdomains.txt"
exit 1
fi
inputdomains="$1"
desktoppath="/mnt/c/Users/stiglmmi/Desktop"
outputcsv="$desktoppath/Ergebniss.csv"
echo "Domain;IntX?;dig-Ergebnis" > "$outputcsv"
while IFS= read -r domain; do
result=$(dig +short NS "$domain")
# Debug-Ausgabe
echo "hdaten für $domain"
echo "dig-Ergebnis: $result"
# Überprüfen, ob "ns14.net" in den Ergebnissen enthalten ist
if [ "$result" == *"ns14.net"* ]; then
nx="Ja"
else
nx="Nein"
fi
# Ergebnisse in der Ausgabedatei in separaten Spalten anzeigen
echo "$domain;$nx;\"$result\"" >> "$outputcsv"
# Debug-Ausgabe
echo "Ja/Nein: $nx"
done < "$inputdomains"
echo "Skript abgeschlossen. Ergebnisse wurden auf dem Desktop in 'Ergebniss.csv' gespeichert."
​
https://redd.it/18msvkm
@r_bash
Problem with understaning bash while trying to follow the manual on installing Dolphin AI locally
Hello.
I'm running Ubuntu through WSL, and I have to say that I'm completely new to Unix in general. The problem is: I'm currently following a step-by-step manual on how to install Dolphin from here:
https://erichartford.com/dolphin-25-mixtral-8x7b
Specifically, this part:
> git clone https://github.com/ggerganov/llama.cpp.git
> cd llama.cpp
> make -j
> cd models
> # download whichever version you want
> wget https://huggingface.co/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF/resolve/main/dolphin-2.5-mixtral-8x7b.Q5KM.gguf
> cd ..
> ./server -m models/dolphin-2.5-mixtral-8x7b.Q5KM.gguf -c 16384
Everything went smoothly until the last command. The response I get to it is:
> -bash: ./server: No such file or directory
And yeah… when I run ls, I can actually see that there's no such directory. Since the manual is pretty much step-by-step, I'm kind of stumped. What did I do wrong and how can I fix it? Any help, please?
https://redd.it/18m3vf6
@r_bash
red syntax error in vim
Hi,
Working my way through TLCL by William Shotts and just curious about why this line in VIM has a red syntax error. It seem to work.
https://imgur.com/a/4fs4Sj8
if (( ((INT % 2)) == 0)); then
Also, why is there a space between (( (( but not once between 0)), seems inconsistent.
Thanks
https://redd.it/18kwk3s
@r_bash
script to add numbers stored in environment variables
Hello, I have a task assignment as follows:
Write a shell script that adds the two numbers stored in the environment variables `WATER`
and `STIR` and prints the result.
* `WATER` is in base water
* `STIR` is in base stir.
* The result should be in base bestchol.
the script must only contain two lines including shebang, and use no operators such as &&, ||, ;, sed or bc.
the script i came up with, is as follows:
`#!/bin/bash`
`printf "%x" "$((0x$WATER + 0x$STIR))"`
assuming that the variables WATER and STIR are set, i understand that i first need to convert the variables from base water and stir respectively, to decimal and add these conversions.
I then converted the result from decimal to base bestechol by mapping the decimal values to corresponding values in bestechol. i am stumped here... while i did ask someone for help, and got the following result:
echo $(printf %o $(($((5#$(echo $WATER | tr 'water' '01234'))) + $((5#$(echo $STIR | tr 'stir.' '01234'))))) | tr '01234567' 'behlnort')
i have no idea how the mapping was done to `behlnort.` Additionally, testing this against the given test cases works for one testcase and none of the others.
**edit during typing:**
i just realised while asking that the mapping was arbitrary and mapping to `behlnort` was arbitrary and i could just use `bestchol.` i am so excited to solve it.
https://redd.it/18jxx4s
@r_bash
I don't understand what is going on here.
I have a script that parse command line for flags that modify the script behaviour.
By accident, I typed invoking the script on cmd line with "=v" instead of "-v" and for my surprise, looks like its breaks getopts somehow because looks like that none of flags are processed in the `while getopts` loop:
```
....
set -o errexit -o nounset -o noclobber -o pipefail
trap go_exit EXIT SIGHUP SIGQUIT SIGTERM
export LC_ALL=C
...
while getopts "s:d:r:hnv" OPTION; do
case $OPTION in
s)
f_src=1
src="${OPTARG}" ;;
d)
f_dst=1
dst="${OPTARG}" ;;
r)
f_regex=1
regex="${OPTARG}" ;;
h)
usage ;;
v)
f_verbose=1 ;;
n)
f_dry=1 ;;
:)
echo "Option -${OPTARG} requires an argument." > /dev/sdtderr
usage
;;
?)
echo "unknow argument." > /dev/sdtderr
usage ;;
esac
done
# sanity checks
if [ "$f_src" == "0" ]; then
echo "Usage: the source folder must be specified." > /dev/stderr
usage
fi
```
All sanity checks that test the f_whatever=0, succeed (as f_whatever are all equal to 0, but it shouldn't as I am passing -s, -d, -r, -n.... - except instead -v it is =v)
Another question related to `set -o errexit .. trap go_exit EXIT `:
If there is any error during the execution the program simply call the go_exit as intended.
How to I get a feedback of which error generated the TRAP and where ?
https://redd.it/18jnvok
@r_bash
Background Processes kept running after exiting
Hi all, first time poster here. Hope this is the right place. I was ssh’d into a server that I work on, where I was running a few processes in the background like this
- ./script &
I accidentally exited the server, and thought the processes would have ended. But they kept going and seemingly finished. Why is that? Can I trust that everything was done right? Thanks!
https://redd.it/18j0h5u
@r_bash
What sed delimiter to use when all are being ued in my variable?
i have a txt file with a bunch of youtube video titles (alot of witch contain / , # , | ... and pretty much all special chars i can think of) i then need to add that to a second file (at the beginning of the second file) useing sed -i "1s/ ( here goes that video title) /" (the second txt file) . Should i maybe use something else than sed?
To specify my situation: i loop trough all lines in the first txt and if that youtube title does not exist in the second file i add it to the beginning of the second file with sed -i "1s/ $(that line as a varibale) /" second_file.txt
https://redd.it/18iwi2y
@r_bash
How to grep a word where I only know the beginning and end of the word?
Let's say I have a long text and I want to find words that start with a and end with n. I thought I could simply grep -o a*n but this will get me no results, even tho those words exist. I guess grep tries to find a three letter word that is a * and n. What can I use here to express an unknown string in between the a and n?
https://redd.it/18imi6b
@r_bash
TIL to continue too
So I have this wee app (bash function), gsi, which loops through files in a git clone, offering actions on each. And it showed me the dir fred/ today.
I do ignore fred.* files in git, but I don't ignore fred/ dirs, could be intersting stuff in them.
But I still don't want to see them in this app, they're not often inetresting.
So I asked the GPT how to add my own ignores list, and it suggested
declare -a CUSTOMIGNORES=("fred" "." "*.cd")
for file in $(gitstatuslinedirchanges); do
[ -f "$file_" ] || continue
git check-ignore -q "$file" && continue
for ignore in "${CUSTOMIGNORES@}"; do
[ "$file_" == *"$ignore"* ] && continue 2
done
done
I've been writing bash for 30+ years, I never knew you could continue 2.
HTH you next week, ...
https://redd.it/18if7qa
@r_bash
ffmpeg and stdout vs stderr
Hi there...
I am aware that ffmpeg outputs everything on screen to stderr. I know how to make it output to stdin, but what I actually want is only the progress stats to output to stdin. Does anyone have an idea on how to accomplish this?
https://redd.it/18i8zpn
@r_bash
Hacky bashy way to restart a server when frozen?
Whenever a tomcat server inside linux is frozen(by frozen, I mean stops printing logs and gets stuck), I want to restart it, automatically.
What is a bashy way to do it?
https://redd.it/18i1mij
@r_bash
Join 2 Bash Arrays in Json using jq
I have two bash arrays.
COLORS=(blue yellow)
FOODS=(cheese ham)
I would like to use jq to combine the two arrays into json that looks something like this.
{
"blue": "cheese",
"yellow": "ham",
}
Or even this
{
"blue": "cheese"
}
{
"yellow": "ham"
}
If not possible I could make it work this way as well, although this would be less preferred.
{
"key1": "blue",
"key2": "cheese",
}
{
"key1": "yellow",
"key2": "ham",
}
Any thoughts on how to do this? Thanks!
https://redd.it/18hmyuv
@r_bash
Just wrote a quick & dirty Bash script for debugging a TetherFi issue, what do you think?
The following is a quick and dirty script I wrote to help debug a TetherFi issue:
<https://gitlab.com/brlin/tetherfi-debugging-utils/-/blob/v1.0.2/measure-time-between-switching-tetherfi-to-background-and-the-proxy-service-stops-working.sh>
Previously I posted a portion of my script for review and the reception is generally quite well, so here's my second attempt! :3
https://redd.it/18hfyp1
@r_bash
OK, I give up. I am search fail. A little grep help please.
I have a file with mostly a crap ton of lines that start with "http".
There are a few lines that start with an English word. A few of those lines are followed by another line which also starts with an English word.
I'd like to grep, or whatever, and get all of the lines that start with a word and are followed by a line with a word, but I only want the fist line displayed in result. eg..
heading
subheading
http..bla...
http..bla...
http..bla...
subheading
http...bla
heading
subheading
http.. bla..
I just want all of the heading lines.
https://redd.it/18guyqu
@r_bash
