Sept 19
Pull Requests
Dockerfile
FROM ubuntu:21.10
maybe we can use Debian instead of Ubuntu
Vroom fetching done, not parsing
Sept 29
Errors
Vroom
https://carsalesystem.cf1.vk.ai/system/batch-review/119922
Blank Parse Statuses Found 1 blank parse statuses, 0 allowed
Cars.com
https://jira.vkportal.com/browse/COL-16579
https://www.cars.com/dealers/14607/mccluskey-chevrolet/inventory/?page=1&stock_type=used
Regex error:
Found 16 errors, 0 allowed
File "/projects/carsalesystem/src/django-vehicleapp/vehicleapp/spiders/cars.py", line 258, in process_dealership_vehicles
num_vehicles = re.capture(r'\d+', matches, required=False) or '0'
TooManyCaptureException: 'Too many matches found (num_vehicles)'
1,118 matches
Evolus
https://jira.vkportal.com/browse/COL-14892
File "/home/code/system/src/django-locationapp/locationapp/spiders/12WestCollection/evolus.py", line 220, in process_specialist
TypeError: expected string or buffer site_id=site_id,
https://locationsystem.cf1.vk.ai/system/group/3/source/991/fetch-batch/200281/parse-batch/205666/
Phone:
- [ ] ‚Äã‚Äã‚Äã(212) 570-2067
Errors:
https://locationsystem.cf1.vk.ai/system/group/0/source/991/fetch-batch/200281/parse-batch/205666/filter/?process_status=3#
Recursion (mark these as bad in code?):
- [ ] https://www.evolus.com/specialists/wendie-do/
- [ ] http://indirect.me/?url=https%3A//www.evolus.com/specialists/taily-alonso-msn-aprn/
- [ ] http://indirect.me/?url=https%3A//www.evolus.com/specialists/tara-howell-md/
- [ ] https://www.evolus.com/specialists/shqipe-rn/
- [ ] http://indirect.me/?url=https%3A//www.evolus.com/specialists/melissa-austin-md/
- [ ] https://www.evolus.com/specialists/elena-ginsberg/
- [ ] https://www.evolus.com/specialists/david-weitzman-md/
Jpg?:
- [ ] https://www.evolus.com/specialists/laura-walsh-pa-c/
- [ ] Etc... maime moose?
- [ ] If resoles to jpg, ignore
Comp
Jason said to comp my monitor in Netsuke (see email)
Oct 5
Eric’s Team Meeting
Matt Daye - lead Docker dev w/ Spencer
S3 coming offline to save $50k/mo
CF-new-generation
Gemini is a VK product of egress points (VPN)
Revisions
Vroom
add missing link-state status
https://github.com/verticalknowledge/cf1-carsalesystem/pull/219
https://carsalesystem.cf1.vk.ai/system/group/0/source/137
https://jira.vkportal.com/browse/COL-16885
Oct 17
Week to talk to Eric
Things to work on:
- [ ] Async
- [ ] Documentation: Sphinx,
- [ ] Logging ??? What is collectionframework.models.LogEntry
- [ ] Testing: method return types ??? What is test_these_links, test_random_limit
- [ ] Performance: bottlenecks / tuning
- [ ] Docker
- [ ] Code style discussions
- [ ] Slack channel for codes snippets: shell, python, perl, rust, go
- [ ] VK University
- [ ] VK Blog (external / public)
- [ ] VK TV
- [ ] Cost analysis: $50K/mo S3 bill
Horizontal alignment
Nesting
ls = self.create_link_state(url=self.url, name='Video Gaming Reports')
self.process_reports(ls)
Vs.
self.process_reports(
self.create_link_state(
url=self.url,
name='Video Gaming Reports’,
)
)
Oct 21
Talked to Eric
Give it 2 more weeks in the fires
Monster in prod
Accidental sql queries to cf1 instead of cf3
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST
kill query 130461309;
kill query 130462218;
kill query 130462421;
kill query 130462461;
kill query 130462532;
Nov 28
PR Duty
Helped Svetoslav with his MR
IT would be good to be able to see for a spider:
- [ ] Is it part of RCS or manual?
- [ ] When is the next schedule? (RCS spiders don’t reveal this in the UI)
Jan 18
Jason 1-1
I need to communicate early on my tickets as I’m approaching ballparks
https://www.zenrows.com/blog/bypass-cloudflare#what-is-cloudflare-bot-management
Читать полностью…
Vertical Knowledge
Paylocity
Company ID: 45540
Username: stevenalmeroth
Password: Stavros13
IT Services / Consulting
Chagrin Falls, Ohio
150 employees
Founded: 2006
Enabling companies and organizations to harness the power of Publicly Available Information
https://vk.ai/
As the saying goes, when everyone's digging for oil, it's good to be in the drill business. Data is today's oil and VK builds the most sophisticated and powerful drilling technology in the market.
harness PAI (Publicly Available Information, also known as public data) - the fastest-growing type of alternative data in the world
Senior Software Engineer
https://jobs.lever.co/vk/6bbea41b-4bea-4832-ae23-47ea08903909
Full-time remote
As an experienced senior level professional, you should be highly results oriented and comfortable assuming a lead role to other junior level colleagues, and being on client facing calls and communications.
Required:
Expert Python
Front-end web (CSS, HTML5, JavaScript, React)
SQL, Elasticsearch and/or NoSQL
Cloud services, preferably AWS
Skills:
Agile
team lead
client calls
documentation
Thursday, July 14, 2022 10:13 AM
Gino Conte
Talent Acquisition Partner
gino.conte@vk.ai
M: 216.374.3319
https://www.vk.ai
finance
spider/bot scrape PAI then sell it
gov spec.ops DOD to stay anonymous "support the troops"
U.S. DoD "Top Secret" Clearance
Wednesday, Jul 20, 2022 11:00 AM Eastern Time
Dan Kasmierski - Project Manager
Interview with SWE III/Sr. SWE candidate Steve Almeroth
Join ZoomGov Meeting: https://vk-ai.zoomgov.com/j/1602320262
"kaz" engineering team 5-10, collection team 20, devops 15-20
tech interview
what are my questions for the team?
5-YEAR PLANS FOR COMPANY
how much time on phone
Dan sounds real
Fosters trust culture
soldiers lives on the line
VK in growth phase, they need the bodies, pipeline is 10x from a few years ago
retool unification collection strategies
stepping from startup to mature mentality
Next: jes & erik partial tech interview
Then: exec interview
great interview!
Wednesday, July 27, 2022 3:00 PM
called Gino to verify it will be a tech interview
Eric Mocny: Director of Collections, Sr software dev, at VK 11 years
collection is a gateway role, can move to other depts after 6-mos or so
eric came from web-design and started when VK had only 12 people (moving from perl to python)
many branches off the vk tree
VK is evolving right now into something new: emerging out into the world
looking to scale
CF1: Collection Framework 1 - in-house Django framework, amazon ec2
CF3: new, the future
CF? - something new new
pycharm
docker
wk1: training spiders
wk2: writing spiders
custom collections, not a data wholesaler
bot detection
1-wk sprints
my experience is rare: I have many tools
team lead role?
now fully remote
still a cool office in CFalls
Friday, August 5, 2022 10:00 AM
Aaron Smith, VP of Collections
been at VK since beginning with Matt Carpenter CEO
aaron putting together a team
gave backstory
clearance - people come safe to their families
my careers - my path aligns well with vk's
mentorship
questions
Turnover
decided not to sell out the soldiers
loyalty
5/6 people turnover
Soldiers get to go home
"shoe spider"?
POI system
keep soldiers alive with video face recognition
my neew challenges
r & d / exploratory
anti-bot measures
q4 2022 - what does the next 10 years of collections look like?
mentorship
frameworks
- based on Scrapy
-
-
remote
pre covid 80% worked in CF
family
won't miss dinners
John CTO
Friday, August 19, 2022 11:11 AM
cameron or clayton are tech guys: ask about using my desktop at home
Santa Cruz Blur TR Fox Performance 120mm/115mm SRAM GX $5900
Читать полностью…
Date/Time: Jul 22, 2022 1:00pm-1:45pm EDT
Interviewers: Anil Kumar: Software dev manager
Google Meet: https://meet.google.com/hhf-ubmf-mux
Date/Time: Jul 22, 2022 2:30pm-3:15pm EDT
Interviewers: Cole Huntley, Amanuel Zeryihun: peers
Google Meet: https://meet.google.com/xmr-tram-xji
Date/Time: Jul 25, 2022 5:00pm-5:30pm EDT
Interviewers: Ajay Khoche: CTO
Google Meet: https://meet.google.com/meu-dwwh-ymg
Class
Wilson's feed mill
Copley feed heritage or heirloom seeds
Chicken needs food grit calcium
Hang roadkill on stick and maggots drop off for chickens
Talk to produce manager or store manager for saving produce waste from trash and donating
Myers hatchery $4/bird / TSE discount birds $1 / Rural King farming equipment also big store
Rainbow Ranger meat birds
Guinea hens good foragers so not great for roosting so better in a cage
Colostrum seals the gut
Ohio farms deficient in magnesium , test soil
Larry Ringer ? Earth foods in Heartville organic fertilizer Holmes ish county
Mosquito killer barn swallows daytime bats nightime
Salt brine chicken in fridge for 2 days
Then salt and buttermilk soak before roasting for crispy skin
Sept 21
Dillon
Helped with retail2 act
https://jira.vkportal.com/browse/RCT-402784
For some reason it looks like PyCharm cf1-locationsystem has caches for the collection framework package
Worked til 11:00 PM finishing Planet Fitness report generator
Oct 1
Saturday
Deep Thoughts
Name
Framework needs a name: like “GrabberWorx”
Walk-thrus
Would be nice to have a walk-thru of component process of:
- [ ] framework - spiders: start, process-methods, reports etc… tasks
- [ ] Docker - Dockerfile, compose etc
Screen Time
Each person shares screen and team debugs for:
- [ ] PyCharm debugger
- [ ] Docker setup so IDE sees framework packages (good on cf3, not cf1)
Oct 14
Notes
Proxies are http (not https)
http://80301:twpfpb@connect.vk.ai:6501
The codebase is ugly, sorry to say
Eric
Im booked this afternoon, but I want to chat next week. There are a few things I want to cover. We cant keep up this pace. Apple for example, we told the client it would be done 30/Sep/22, its 14/Oct/22 and is not even on track to go out today.
1:37
On track to deliver 0 COL tickets this week
1:38
So I’d like to understand what you are doing with your time, because there are only ~20 hours logged between those 2 tickets, and like I said, one has been open for 2 full weeks.
Oct 24
Refinitive
Training presentation by Scott Jewell
Increasing client
Finance sector
Running on super old data but it’s our job to deal with it
Sharepoint / one note
Web-grabbing
Email-grabbing
Ftp-grabbing
2019
Spiders should use Scott’s RefinitiveBaseSPider
RIC : Reuters Instrument Codes (rows description)
FID : Field Identifiers (column heading)
Value: table cell with the data
CSV file
pre-formatted
JSON file
format data
uploaded to S3, inserted into their db, removed from bucket
doesn’t actually need to be strict correct JSON, we can use ”val”: 7.5000
We probably don’t need to write a generate_reports method, just use the one in base class
RefinitiveEmailBase
fake link_states
superseded by S3FetchBase
S3FetchBase
not real link-states
everything handled in start method with helpers
errors are hard to find as link-states don’t report them
sort_s3_messages task kicks off jobs, not schedule
do not create fold in S3! Everything is a flat file and Cyberduck will create some random file
add new collections to this file
Keep LAST_UPDATED_DATE spider setting
PROD_TEST_MAP
Nov 22
Jason 1on1
I need to do better with Attention to detail: I gave an xls report when they wanted csv
Keep SD in the loop: don’t spend 3 hours on something that maybe SD doesn’t really need
Jan 30
Company Meeting
Brian O’Keefe announcing
John (JPK) moving from CTO to advisory role, Aaron (Sharrow?, not Smith) to fill his shoes
Food Storage
8 x Patriot Pantry 2-week buckets @ 2000 calories /day /person
3 x 88 servings protein: 32 meat, 56 beans
Trek Top Fuel 9.7 Fox Performance 120mm/130mm Shimano $4500
Читать полностью…
HD
JB weld
Supply turnouts (purple) 3
Turnout brace 2
Pex valves 5
Escutcheon 5
1/2" round KO box cable conns
CRAWL
organize clothes
camera
Red cable truck
USB adapter
Toiletries
Mouth guard
Farm
0 Tomato
1 lettuce 🥬
2 carrots
3 dill
4 dill
5 turnips
Squash
cukes
Frijoles
