News
Source:news.md
MAPS 2.2.2
Date: 2026-05-19
- fixed bug that dropped discard estimates on the last day of the year for calendar FY species
- fixed inconsistency in national bycatch report output table between “fed” and “federal”
- added large and small mesh_cat to the twin otter trawl gear in cfg_nbr_fleet. OTT had a null mesh_cat in the original nbr report, but has a mesh_cat in CAMS
- added explicit delete of discard output fst files during internal process to prevent rare duplicated estimates from zombie files
MAPS 2.2.1
Date: 2026-05-13
- fixed EM delta model bug to use year as a random effect rather than a fixed effect for FY 2026+ to prevent failure when no EM reviewed trips yet in the year.
MAPS 2.2.0
Date: 2026-03-13
Column name changes:
- Added columns VTR_LBS_DISCARD (replaces VTR_DISCARD), VTR_COUNT_KEPT and VTR_COUNT_DISCARD to tables CAMS_LAND, CAMS_VTR_ORPHANS and CAMS_VTR_REC.
Discard changes:
- Fixed a bug where non FMP species in the FOSS_NBR_ALL_SPECIES view did not link to a fleet in CFG_NBR_FLEET.
Landings changes:
Fixed a bug with CAMS_VTR_REC where the count of fish was incorrectly labeled as LNDLB.
For scenarios, where fish were sold on a rec trip and are found in CAMS_LAND, the table now includes the weight AND count of kept fish (VTR_COUNT_KEPT).
Lobster Management Areas added to CAMS_SUBTRIP (AREA_LOBSTER)
MAPS 2.1.3
Date: 2025-12-01
- Changed bluefish discard mortality rate from 100% to 32% for all gears to align with 2022 research track assessment: https://repository.library.noaa.gov/view/noaa/48948
- Fixed groundfish binning rules for fishery group to prevent any DOF/undeclared recreational trips from being assigned to groundfish
- Added Mnk-E to permits allowed for a mispecified groundfish trip
- Removed extra whitespace from HULLIDs so seemingly blank cells are treated as missing rather than ” ”
- Fixed a bug for EM trips with unknown stat areas to inform review rates (effected 4 trips in 2025)
MAPS 2.1.1
Date: 2025-09-19
- Fixed bug introduced in 2.1.0 affecting windowpane and yellowtail flounder discards on non-groundfish trips
- Added internal processing to improve testing
- Added process to share
STG_OBDBS_ALL_YEARSwith APSD to allow for use in quota monitoring
MAPS 2.1.0
Date: 2025-09-15
Column name changes:
- CFLEN_AA.SPECIES_ITIS to CFLEN_AA.SAMPLE_SOURCE_CODE to better reflect data source
- Three columns changed to ensure same name for same data:
- CFG_STATAREA_STOCK.AREA_NAME to CFG_STATAREA_STOCK.SPECIES_ESTIMATION_REGION
- CFG_DISCARD_MORTALITY_STOCK.AREA_NAME to CFG_DISCARD_MORTALITY_STOCK.SPECIES_ESTIMATION_REGION
- CAMS_DISCARD_STRATA.SPECIES_STOCK to CAMS_DISCARD_STRATA.SPECIES_ESTIMATION_REGION
- CAMS_ALL_PTNS_EM_TRIPS.FMP to CAMS_ALL_PTNS_EM_TRIPS.PTNS_FMP
- Added column CFG_FED_AREA.AREA_BROAD_STOCK to improve delta model transparency and data management
- Added column CAMS_OBS_CATCH.MESH_RECORD_TYPE to provide transparency on any assumptions made to missing data on observed trips. These assumptions allow for all observed trips to be used for assigning observed discards to trips rather than applying a discard rate when mesh is missing.
- CAMS_OBS_CATCH.CAMS_SUBTRIP to CAMS_OBS_CATCH.SUBTRIP removes concatenation of CAMSID and SUBTRIP and maintains indexing across CAMS tables of CAMSID and SUBTRIP
- LOBSTER_CLASS column changed to PERMIT_CLASS in 5 locations:
- CAMS_SUBTRIP
- CAMS_VTR_REC_SUBTRIP
- CAMS_AQUACULTURE_SUBTRIP
- CAMS_CFDETT_AA
- CAMS_LANDINGS (GARFO view)
Discard changes:
- Ensured that EM groundfish trips that are changed to non-EM after PTNS reporting do not get delta estimates and instead get discard rates.
- Updated delta model to use the fishing year as a random effect rather than a fixed effect, and to use a median Q50 rather than an estimated average for the hurdle lognormal model.
- Updates to skate mortality ratios based on best stock assessment information.
- Ensure observed discards are used on observed trips, even when missing mesh size information, rather than getting a discard rate.
- Added assumption of large mesh category to observed trawl and gillnet hauls with unrecorded mesh on trips with multiple subtrips . This prevents observed discard data from being dropped when mesh information is missing.
- Ensure fishing year 2025 EM trips use the delta model where appropriate.
- Changes to discard mortalities based on observer discard reasons; discards marked as: Previously Discarded, Discarded by Observer, Intended Catch, or Discarded due to Vessel Casualty, Intended to be Kept will not contribute to general discard rate estimation.
Landings changes:
- Ensure assignment of VTR orphans does not lead to duplicated CAMSID in CAMS_LAND and CAMS_VTR_ORPHANS.
- Increased transparency by adding a DATE_RUN column to support tables.
MAPS 2.0.8
Date: 2025-09-02
- Added precaution to prevent duplication of link1 between observer and prelim observer data. It was not supposed to happen but a bug in the new compass ob system allowed duplication. This will make CAMS more robust to those errors.
MAPS 2.0.7
Date: 2025-08-18
- Internal updates to transition to OCI cloud databases and app servers
- Updates for R and package versions associated with cloud migration
- Switch to writing tables directly to garfo production database so cams database can be turned off when not running. Then TTS from garfo production. Will greatly improve speed at GARFO and have no effect on NEFSC. Update connections and DBlinks. Dropped stage tables from being available on garfo production.
- Switched website location and associate code to https://apps-garfo.fisheries.noaa.gov/cams/ on cloud web app server
- Fixed bug causing some VTR trip orphans to go into cams_land when there were vtr discards on those trips.
MAPS 2.0.6
Date: 2025-06-09
- Extends all tables with ITIS_TSN to hold 11 characters to meet current regional standards with left padding to 6 characters. Some tables still only allowed varchar(6) rather than varchar(11). At least one upstream record reporting a 7-character ITIS code so this was critical for CAMS to ingest the upstream data.
- Bug fix for groundfish fishery group to include undeclared/unmatched declaration trips to be assumed groundfish in addition to DOF if they have MUL-A permit and landed >5% groundfish. This prevents them from going to uncategorized fishery group.
MAPS 2.0.5
Date: 2025-06-04
- Fixes a bug that was causing a mismatch in the accsp_category_code for scottish seine (negear 360) between VTR and EM (NEMIS). Changes it from 110 to 100 in cfg_negear.
- Expanded
cfg_stock_assessment_areastable: Addedassessment_typecolumn with default STANDARD but also TRAC for cod and haddock. Also expanded to include all Atlantic federal statistical areas for each species-assessment_type combination with OUTSIDE for those areas not used in the assessment. AREA = ‘000’ is not included in this table and will be the only NULL values when joining with CAMS_LAND (rare cases of fishing from shore from state locations without a specific port or county). Also added sys_guid primary key to start better data management and architecture practices.
MAPS 2.0.4
Date: 2025-05-28
- Bug fix for edge case groundfish trips to get reported area over carea. Very little effect to stock assignments. For example, only 8,000 lbs of cod changed stock area as a result for ALL years from 1996-2025.
- Bug fix for lobster trip area assignment. Trips after 2024-04-01 were still getting the adjacent port area assigned in most cases due to a bug. They will now use the VTR areas as appropriate following the CAMS documentation as intended.
- Split up functions for efficiency and improved troubleshooting.
- Added 22 unit tests to improved development
MAPS 2.0.3
Date: 2025-05-05
- Changed source table for sector rosters from
SECTOR.SECTOR_YEAR_ROSTERtoSECTOR.MV_MRI_PERMIT_VESSEL_HISTORY - added a function to help with updating versions so don’t miss dropping tables or views with DDL changes and can be more fully tested in preproduction. No effect on end users, just for ease and transparency in development.
- Updated
vtr.vlporttbltotrip_reports.vlporttbl
MAPS 2.0.2
Date: 2025-04-11
- Bug fix for as.Date implicit conversion to UTC which caused errors in DA calculation
- Added non-fmp discard estimation for National Bycatch Report (NBR) including tables
cams_discard_nbr_non_fmp,cfg_nbr_fleet, andcfg_nbr_itis.
MAPS 2.0.1
Date: 2025-03-31
- Bug fix to make the
state_completefield ofcams_landings_metadatamore accurate - Bug fix to prevent vtr_orphan_species duplication of surf clams and ocean quahog when data corrections are made after 2024-11-01. All records with corrections, submissions, or resubmissions after that date go through the trip reports system and go into BOTH
sfclam.sfoqvrand the normal VTR tables and views (CID). The previous CAMS code only included trips landing after this date rather than all data updates after that date. This resulted in <0.75% duplication of clam data for 2022-2024 (none for 2025 or before 2022).
MAPS 2.0.0
Date: 2025-02-15
Major Updates
The only major change that should result in the breaking of well-written code will be to the CAMS_DISCARD_ALL_YEARS table. This table is exceedingly and unnecessarily large with information duplicated from other tables. The size of this table makes processes slow and fragile, it dramatically slows down data sharing, and it makes end user queries extremely slow.
To alleviate these issues, we are dropping all rows where DISCARD_SOURCE = ‘N’ since those are not or could not be estimated and therefore always have zero discards and zero variance. The total number of subtrips and the subtrip Kall will still be available via the CAMS_SUBTRIP table and the total discards and variance (and therefore CV) will still be correctly summable within the new CAMS_DISCARD_ALL_YEARS table. Additionally, we will be removing columns that are unnecessary and found in CAMS_SUBTRIP.
Additionally, we are removing the discard rate and stratification information from CAMS_DISCARD_ALL_YEARS so that it’s not duplicated 100s of times for each row to which the strata rate applies. This information will still be available in the new table CAMS_DISCARD_STRATA and can be joined without duplication to CAMS_DISCARD_ALL_YEARS by the primary key DISCARD_STRATA_DESC.
Description and examples of using the new table structure can be found on https://www.greateratlantic.fisheries.noaa.gov/ro/fso/reports/cams/articles/common_query_examples.html and the entity relationship diagrams (ERD) are now updated on https://www.greateratlantic.fisheries.noaa.gov/ro/fso/reports/cams/articles/relationship_diagrams.html.
Finally, the column VTR_GEAR_CODE in CFG_GEARCODE_STRATA will be renamed to SECGEAR_MAPPED because it was not the actual VTR gear code but rather a gear grouping for matching between data systems. This does not change any process or resulting data but would affect anyone using this column name. Additionally, the values within this column, which currently look like VTR gears, will have the prefix “G_” to indicate that they are groupings and differentiate them from individual VTR gear codes.
We have also dropped the Dropped CFG_MASTER_GEAR table as it duplicated other gear support tables but in a less straightforward way.
The discard tracker was renamed to QC_DISCARD_TRACKER to be more inline with other naming conventions and the full discard strata column was dropped as it was not relevant. The previous version from NEFSC that was run by Susan Wigley was combined with the production one in CAMS to have a longer time series. The process was also adjusted so that a complete grab of all discards is no longer done each week. Now, only discards calculated or recalculated since the last grab data are inserted into the table. This prevents duplicate information and the excessive growth of the table.
Minor Updates and Bug Fixes
- Added column ACCSP_CATEGORY_CODE from CFG_MASTER_GEAR to CFG_NEGEAR which is primarily used to match EM records to VTR records.
- Fixed a bug with non GF gears on GF EM/Audit model which were incorrectly getting a delta discard estimate instead of a discard rate (e.g., lobster pots, where EM discard handling/methods are not performed).
- Added TRIP_DURATION, TRIP_DURATION_Q25, and TRIP_DURATION_Q75 to CAMS_SUBTRIP to better reflect the time away from dock for the trip. The calculation is date_land - date_sail (including fractional days). The total for the trip is then apportioned across the subtrips in proportion to DF (days fished) so the sum across subtrips equals the total time away from dock.
- GDP deflator did not come back on the first CAMS run after wiping the schema for CAMS 2.0. This was fixed so it comes back when the schema is blank and doesn’t depend on checking the date in the existing table. Previously it would have likely taken 60 days before returning.
- Scallop Area added to CAMS_SUBTRIP
- Added scallop area NYB for the new VMS declaration and area associated with the New York Bight
MAPS 1.1.7
Date: 2025-01-07
Updates
- Updated build of CAMS_GDP_DEFLATOR because of upstream changes in source structure. Added columns for date_start and date_end to improve the ease of joining to other tables.
- Added use-case for the deflator on website quick start guides
- Added example of joining in VTR gear info to CAMS data based on user feedback during office hours
MAPS 1.1.3
Date: 2024-10-16
Bug Fixes
- Fixed incorrect Council designation for spiny dogfish and monkfish.
- Removed blackfin monkfish from being labeled as in the monkfish FMP.
- Added red crab and wolffish as FMP species.
- Corrected definition of the DELTA discard source in cfg_discard_source.
- Carry lat/lon data from SFCLAM.SFOQVR through to end user tables
- Fixed bad date join with SFCLAM.SFOQVR and dealer data
- Carry SFCLAM.SFOQVR.TRIP_NUM through as a DOCID since it is the vessel trip identification code and is a DOCID since clams started submitting through eVTR
- Fixed triplets bug em is NULL and discard_source=DELTA and nemis/cams mismatch with scottish seine accsp gear category
- Fixed duplication of mixed hake vtr_orphan_species
MAPS 1.1.1
Date: 2024-10-08
Bug Fix
- Fixed a place in the EM process where it connected to the preproduction database and therefore had some stale data. The preproduction database was being used in testing regularly so the data were only approximately two weeks behind for one small portion of the delta EM discard process. Comparisons showed minimal difference in all stocks and years. All years of groundfish EM discards were rerun (only groundfish on groundfish EM trips could be affected by this).
MAPS 1.1.3
Bug Fix
Removed EM trips with no VTR from cams_gf_em_delta_vtr_discard because it was triplicating discards for those trips with no VTRSERNO.
Fixed inconsistencies in ACCSP_GEAR_CODE for gear type scottish seine between EM and VTR system that was failing any EM reviewed trips from the one EM vessel using scottish seine.
MAPS 1.1.0
Date: 2024-09-26
Updates
- The CAMS Team implemented methods to impute the statistical area and KALL from offwatch hauls so they can be treated the same as other observed trips where discards are imputed from unobserved hauls. This change primarily resulted in a shift in the discard source from a discard rate (“A”, “T”, or “I”) to actual observed values (“O”) for trips with offwatch hauls, and generally had minimal effects on overall discard estimate amounts that were concentrated in the scallop and clam fisheries. Refer to the CAMS documentation for details of the offwatch haul procedures.
- Added PTLL and PTCL to CFG_VLGEAR to prevent imputation of these new pseudo-gear codes that didn’t previously map to any NEGEAR. Now they map to NEGEAR 200 and 300 as intended.
- Fixed bug so before 2024-04-01 lobster trips consistently use the port-adjacent area and not the dealer-reported area. After that date, lobster trips are handled the same as all other trips with mandatory VTR reporting as described in the documentation.
MAPS 1.0.18
Date: 2024-08-14
Updates
- Fixed handling of missing EFP data from new EFP since they are part of the regular EM fleet. No changes to past data or previous run. Just an emergency fix for a new type of data error for a new(ish) program. The missing value in one row cause a duplication that was caught and halted the CAMS run until implementing this fix.
MAPS 1.0.14
Date: 2024-07-14
bug fixes/corrections:
- Fixed a bug to correctly use fishdisp in OBPRELIM so all appropriate prelim data is used. Previously, not pulling in most preliminary observer data.
- Added handling to assign state in landings tables based on port substring (first two digits) when a port is not in the CFG_PORT table. This is especially true for state trips that have an “unknown” (99) port code. Future work will unify cfg_port with valid upstream port tables from trip reports and ACCSP.
MAPS 1.0.12
Date: 2024-06-12
- Added addendum to CAMS Change Management Plan for clarification about potential new data or models in CAMS
- Updated internal CFG Operations to make an easier process less prone to errors
- Added three species codes
| itis_tsn | sci_name | com_name ITIS_DB_STAT | ||
|---|---|---|---|---|
| 1 | 095638 PENAEUS MONODON GIANT TIGER PRAWN | valid | ||
| 2 | 099080 | OCYPODIDAE | FIDDLER CRABS | valid |
| 3 | 166882 | PTEROIS | BUTTERFLY-CODS | valid |
MAPS 1.0.11
Date: 2024-05-07
Discards:
- Changed to not estimate discards for gear group 0_pottrap_na for butterfish (ESTIMATE_DISCARDS = 0). Experts were confident that butterfish were not being caught in these traps and any observed were a result of data errors
bug fixes/corrections:
- Fixed leap year handling of sector end dates when missing
MAPS 1.0.10
Date: 2024-04-23
Discards:
- Changed Cod mortality in lobster pots (NEGEAR = 200) from 1.0 to 0.248.
bug fixes/corrections:
-
add_nobsfunction: to properly assign numbers of observed trips and unobserved trips according to discard_source -
assign_discard_source: fixed assignment of ‘G’ discard source -
cochran.calc.ss: moved function calls from plyr to dplyr verbs -
get.cochran.calc.ss: fixed problem of large integers creating NA (integer overflow) -
discard_XX: cleaned up broad stock section of code -
discard_XX_diagnostic: cleaned up broad stock section of code, fixed schema.table references throughout
additions:
discard_herring_diagnostic.r get_catch_obs_diag.r
help files and description/namespace changes
MAPS 1.0.9
Date: 2024-04-17
- Scottish Seine gear discards now estimated for groundfish
- ACCSP Gear Category code 110 (scottish seine) was added to the EM gear matching and use
MAPS 1.0.7
Date: 2024-02-07
Updates
Landings:
- Updated CFAGE and CFLEN to point to new versions in BSM on NEFSCDB1
- Added SAMPLE_SOURCE_ABBR VARCHAR2(18 BYTE) and SPECIES_ITIS VARCHAR2(18 BYTE)) so users know the program under which the data were collected
- Added 01 April 2024 as the start of required federal lobster VTR requirements. Therefore, area imputation will be from VTR rather than port-adjacent stat area for trips landing after that date.
Discards:
- Updates to calculation of
n_obsvalues - Adjusted code to explicitly use number of observed subtrips within a
STRATA -
n_obsmay include observed subtrips across fishing years whenDISCARD_SOURCE=TorG - Unobserved subtrips,
N_USED, refers only to the current fishing year. This simply counts the number of subtrips in theSTRATAthat had noLINK1. Previous methods of subtractingN_USED-n_obsdo not work when n crosses fishing years. - added a
add_nobs.Rfunction - edited the
get_covrow.Rfunction which previously did the n_obs step - edited all
discard_functions, including diagnostics - added a
discard_herring_diagnostic.Rfunction
-Added a new table, qa_gf_em_subtrip, that contains information on em video monitoring trips selected for review. This table allows end users to see which em reviewed subtrips passed or failed review based on business rules associated with observed hauls (all hauls need to be reviewed) and percentage of fish unknown to species (<10% when fish count is >30) on em reviewed subtrips.
-Discarded redfish recorded on em reviewed trips are now apportioned between redfish and scorpionfish nk based on a nemis helper function (scorpionfish_species_lbs_ratio). This function was added because em reviewers cannot distinguish between redfish and scorpionfish species during video review.
Process and Operations
- David Weekley created two new tablspaces to facilitate potential transportable tablespace swaps (TTS) which would allow for creation of materialized views from CAMS objects
- Weekley created a procedure that will allow the CAMS_GARFO user to place the new tablespaces: CAMS_SHARED_DATA and CAMS_SHARED_INDEX into either read only or read_write mode. Both new tablespaces will grow as needed.
- The CAMS development team changed all tables with CAMS_, CFG_, MATCH_, and QA_ to be within the CAMS_SHARED_DATA tablespace and the associated indexes to be in CAMS_SHARED_INDEX. All other data and indexes stayed in their previous tablespaces
- The procedure to alter the read-write status of the tablespaces was added to the run_maps.R function at the start and end of the production sections (when data moves from MAPS to CAMS_GARFO schema)
MAPS 1.0.5
Date: 2023-12-14
Updates
- Added FY and FY_TYPE to STRATA_USED and STRATA_USED_DESC in the CAMS_DISCARD_ALL_YEARS table for complete strata definitions CAMSNR-727
- Switched yellowtail and windowpane on scallop trips from being 365-day lookbacks to being on the FY so matches all other discard estimation processes CAMSNR-727
- Fixed bug for unobserved subtrips on observed trips: CAMSNR-736
MAPS 1.0.2
Date: 2023-11-04
Updates
- Added hullid to CAMS_ALL_PTNS_EM_TRIPS and also stopped using ptns_report_compliance_mv as it was overly restrictive. This table is used by FMRD in production and the updated code was developed by Brant McAfee and Daniel Hocking. No other code in CAMS is dependent on this table.
- Discards: changed link3_obs to partition at the subtrip level rather than the trip level.
- Corrected the exempt_7130 to be at the trip level to match the regulations (the whole trip has to be west of 7130 to be exempt)
MAPS 1.0.0
2023-10-11
Updates
- Removed annual CFDETTyyyyAA, CFDETSyyyyAA, CFAGEyyyyAA, and CFLENyyyyAA tables so only the *_AA version with all years starting in 2020 get produced (2020 - 2023 for CFDETT/CFDETS, 2020+ for CFAGE/CFLEN)
- Monkfish trips in the MONKFISH column in CAMS_FISHERY_GROUP are now inclusive of the mixed monkfish/groundfish trips. This does not affect the overall primary FISHERY_GROUP column
- NEFSC CAMS working group core team members now will access staging tables and other non-primary production tables and any testing tables via their individual schemas on GARFO’s production Oracle instance.
MAPS 0.9.8
2023-08-28
Updates
- Fixed bug in groundfish scallop subroutine that caused incorrect values in the end year (2023) of discards for the two flatfish species. The cause was differences in Groundfish and Scallop fishing years while looping through groundfish fishing years resulting in a mismatch in the final loop.
- Excluded research trips from the misdeclared DOF and assumed_ground columns in CAMS_FISHERY_GROUP. Does not change the overall primary fishery group since those rules were already in place. This just alleviates potential confusion.
Process update
- Added parallel processing to discard calculations, discard uploads, and the push of tables from MAPS to CAMS_GARFO for improved runtime speed
Website
- Removed the review comparisons for now in prep of releasing the documentation on a public website for council staff and sector managers to view. This removes any potential confidential info.
- Adding more code examples to the quick start guides
- Adding more details to column and table descriptions in the data dictionary
MAPS 0.9.7
2023-08-04
Updates
- Added cams_version to allow users to determine what cams version was associated with each record date_run
- changed CFG_VTR_ITIS BAIT, OFF, and PELto map to mapped to 914179 (Gnathostoma) rather than to 161030 (Osteichthyes) to reduce orphans and be more accurate
- In CFG_SRCE_ITIS changed 161030 should to map to the slightly more specific 161061 (Actinopterygii) rather than Ganthostoma since it’s fish and not sharks/skates
- Added disposition 13 handling to the documentation
- Modified BSB discard mortality rates from 1.0 to 0.15 for all cams_gear_groups except gillnet and trawl
MAPS 0.9.4
2023-06-23
Updates
- Stopped imputing area, gear, and effort for non-fishing trips (vtr_activity > 0). We shouldn’t guess where they were or what gear was on board if they weren’t fishing with it. And there was no effort, it doesn’t really apply to non-fishing trips (mechanical turn-arounds, etc).
- Separated groundfish STATE and UNCATAGORIZED fishery groups
- Renamed OTHER2 fishery group to GROUND_ASSUMED
- GF now includes GROUND and GROUND_ASSUMED
- Ensured that permits start at the beginning of the data and end at the end of the day. Important for assigning fishery group, including groundfish, which then influences discard estimation for trips that land the day permits end or start.
MAPS 0.9.3
2023-06-16
Updates
- Fixed and vs or statement in rolling join to avoid duping or dropping during staging
- Fixed dealer combined trips code to avoid dropping VTR from matched dealer records during staging and matching processes
- Added vessel information from SFCLAM to dealer source 08 to get lat/lon and areas for best source
- Added cfg_scallop_area for transparency and end-user convenience (no effect on processes)
- Fixed estimation 0/1 to be consistent within a CAMS_GEAR_GROUP for a given species
- Added Fish nk% to EM table to allow for sector/vessel transparency with respect to failed review for inclusion in delta model or direct EM discard value
MAPS 0.9.2
2023-05-24
Updates
- Renamed CFG_ITIS to CFG_SRCE_ITIS because the primary key is SRCE_ITIS_TSN and end users would have difficulty using it for ITIS names without causing Cartesian products during joins
- Made new CFG_ITIS as a distinct derivative of CFG_SRCE_ITIS without the three SRCE_ columns so the primary key is ITIS_TSN. This is automated in the code so that only CFG_SRCE_ITIS needs to be maintained and the other will stay in sync.
- Fixed minor discard_source labeling bug
- Recalculation of N_OBSERVED and N_UNOBSERVED to represent number of observed subtrips used in the CV estimation (can be from multiple years or the previous year depending on discard source) and the number of subtrips that rate was applied to in that year, respectively
MAPS 0.9.0
2023-05-07
Updates
- Reduced the size of the discard table and restructured it to be more efficient and user-friendly. The change reduces backwards compatibility (i.e. breaks end-user code). Pre-final production this is a “minor” semantic version change (0.x.0) but post-finalization this would be a major change in the semantic version (x.0.0).
- Reduced size of
CAMS_DISCARD_ALL_YEARSby removing several columns, the most notable of which is theCAMS_SUBTRIPcolumn.CAMS_IDandSUBTRIPremain, and both must now be used to join to other tables. - Added several columns:
N_OBSERVED,N_UNOBSERVED,CAMS_VARIANCE,STRATA_USED_DESC -
CAMS_VARIANCEis the most notable addition as this is the covariance of the discard estimate, at the subtrip level. Credit to Chris Legault for the process to make this a reality. -
STRATA_USED_DESCcontains the individual elements for the stratification used on the subtrip.
MAPS 0.8.0
2023-04-18
Updates
Discards
- Added implementation of the ESTIMATE_DISCARDS columns such that any non-observed discards get multiplied by this column (zeroed out if estimate_discards = 0). DISCARD_SOURCE = ‘N’ for these records.
- SNE yellowtail set to 0.9 for all gears per the SARC 54 Assessment Summary Report.
- Halibut, set some selective trawl gears to 0.76 and “0_gillnet” to 0.3 per the 2017 Assessment.
- GOM winter, set to 0.5 for all gears per emails with assessment scientists.
- SNE winter, set to 0.5 for all gears per the SAW 52 Assessment Report
- Modified gear stratifications for Alewife, Blueback herring, River herring, American shad and Hickory shad.
MAPS 0.7.1
2023-03-31
MAPS 0.7.0
2023-03-16
MAJOR UPDATES
Updates
Landings
- Separated VTR Orphans from landings into CAMS_VTR_ORPHANS and CAMS_VTR_ORPHANS_SUBTRIP
- Reclassified vtr trip orphans with discards or specifying no catch as status VTR_DISCARD and VTR_NO_CATCH respectively so they go into cams_land/subtrip for trip and effort accounting. They were not real orphans since no sale was expected.
- Fixed NA handling causing extra live pounds (but not landed pounds) for monkfish and skate parts
- Ran landings back to 1996
Discards
- CAMSNR-358: Modified surfclam and quahog stat area stratification. Areas 521, 526, 533, 534, 537, 538, 539, 541 were changed from NE to MA.
- CAMNSNR-106: Revised atlantic herring gear groups to four major gear groups (MW, BT, PS, Other).
- CAMSNR-561: Modified golden and blueline tilefish area stratification. Areas 525, 526, 533, 534, 537, 538, 539, 541, 542, 543 and 562 were changed from NE to MA. Also, added 631, 632, 634, 635, 636, 637, 638 and 639 to MA.
- CAMSNR-562: For witch flounder (172873), changed PTS from 0_pottrap to 0_pottrap_na in cfg_gearcode_strata.
- CAMSNR-555: Changed negear 150 from VTR gear code OTF to OBP in CFG_NEGEAR.
- Changed groupings of otter trawl gears to match stock eff groupings with spiny dogfish, monkfish, golden and blueline tilefish and skate species.
MAPS 0.6.4
2023-02-17
Updates
- Added DLR_STID (state dealer number) and DLR_CFLIC (State Commercial Fishing License) to CAMS_LAND and CAMS_LANDINGS
- Added VTR Mesh cleaning using CFG_VLGEAR to make sure reported values fall within the min-max range
- Throughout Discards: MESHGROUP -> MESH_CAT
- CAMS_OBS_CATCH = ACCESSAREA and TRIPCATEGORY (“all” -> “ALL”) - attempted but incomplete/ineffectual?
MAPS 0.6.3
2023-02-08
Updates
- Dropped old support tables that started with CAMS_ (e.g. CAMS_MASTER_GEAR) since all code should have already been switched over to the CFG_ version (e.g. CFG_MASTER_GEAR). This is to prevent uptake of duplicate support tables that are no longer the supported version.
Bug Fixes
- Fixed timezone issue to avoid offset pushing trips to the next day
- Fixed duping of 2022 calendar year discards introduced in v0.6.2 when attempting to fix the empty observer data table bug with the 2022/2023 switch
- subtrips with >1 LINK1 had the OBS_DISCARD duped for each row. Normally, there is only one row/subtrips in CAMS_DISCARD _ALL_YEARS. This is a fringe case. Solution: OBS_DISCARD, that is, the observed discard values for the subtrip, are added in a different place in the code. OBS_DISCARD is not added during the run_discard.R function. This allows for an appropriate join without cartesians.
- discard_herring.R was still using VTRSERNO as a grouping variable. This was replaced with CAMS_SUBTRIP and now matches the other discard functions.
- added allocated vs non-allocated flag to discard_groundfish.R
- added logic to adjust MREM trips for allocated groundfish according to rules (assumed discard = 0)
- added a new DISCARD_SOURCE = ‘R’ (rule-based assignment)
- added description for new discard source to CFG_DISCARD_SOURCE
MAPS 0.6.2
2023-02-01
Updates
- Forced groundfish trips to be at the trip level rather than the subtrip level to match legal definition. It’s unclear if there were any cases where a trip was mixed given the current definition but it could happen in theory with the old code.
- Fixed bug that caused GF = 0 in CAMS_DISCARD_ALL_YEARS for herring discards on groundfish trips (GF was still 1 in CAMS_LANDINGS)
- Changed permits “190998”, “290998”, “390998”, “490998” to user the same CAMSID structure as permit 000000 to better differentiate trips that were recorded on the same date in old data
- Added one more exception for updating a invalid ITIS code from 2001 in CFG_ITIS
Testing
- Tested running CAMS_LANDINGS back to 1996. Currently info only on GARFO side in CAMS_LAND_1996, CAMS_SUBTRIP_1996, CAMS_LANDINGS_1996. Production is still 2005 - 2023. 2005 was an abitrary stop previously in part because of issues with older xxx998 permits regarding trip/subtrip splits. Will discuss with CAMS team whether to review older data or just move it into the main tables.
MAPS 0.6.1
2023-01-25
Updates
- Added CAMS_SIMM_DISCARD as a view created by D. Caless of observed discards formatted for the SIMM download and granted it to SECTOR and APSD
Process
- Updated calendar year and herring discard function to work early in the year when there is no observer data yet (no longer fails because of returning an empty dataframe before looking to the previous year). Still need to do it for groundfish but have until May 1 before the next FY
- Removed old discard module functions to prevent confusion. They can still be viewed in previous commits if needed.
- Moved landings log from local to production output location with the frontend log.
- Added discard log rather than screen printing
- Added production.log to log the push to CAMS_GARFO, documentation build, and website builds.
MAPS 0.6.0
2023-01-05
Updates
- Added 2023 data
- Included 3-month previous-year buffer in estimate of area level “E”, gear/mesh level “C”, and mesh level “I” imputations. The purpose is to be able to roughly impute area, gear, and mesh even during the first weeks of a new year.
- Switched EM Delta model to hurdle-Poisson (from just Poisson) and using the median estimate to better handle zeros and keep predictions from unrealistically high values. Hurdle-lognormal could be used to be 10x faster but would require sensible priors to avoid high variance at larger values. It works well but not as well and has a slight negative bias with unrealistically large, vague priors.
MAPS 0.5.1
2023-01-04
Updates
- Separated CAMS_LAND and CAMS_SUBTRIP then made a view of CAMS_LANDINGS without DF and DA info. CAMS_LANDINGS will be very slow across DB links compared with using CAMS_LAND and CAMS_SUBTRIP directly.
- Fixed NA handling BUG with MREM for pre-2020 data in discards
- Fixed discard duplication bug associated with some MREM trips
- Added CAMS_ALL_PTNS_EM_TRIPS for defining FY20+ EM trips using CAMS matching
MAPS 0.4.2
2022-12-15
MAPS 0.4.1
2022-12-14
Updates
- Fixed bug where trips with multiple AMS DASID on a single trip were getting duplicated in landings through GF/FISHERY_GROUP joining. This bug was in the data from approximately 23 November - 11 December 2022.
- Added EXEMPT_7130 (0/1) to CAMS_SUBTRIP. It is always zero before 2022-12-15 which is the effective start date for this portion of Amendment 23.
- Added 71-30 exception to groundfish stratification
- Updated EM delta model to use median posterior predictions from a hurdle-lognormal distribution to prevent the occasional excessively large (unrealistic) discard estimate
- Updated the ITIS_GROUP1 & ITIS_GROUP2 to all use “WHELK” for these rather than distinguishing between CONCH and WHELK when joining DLR and VTR since it’s too easy to mis-ID. This is in response to a message from DIS: “NE observers have been instructed to record WHELK, NK, CONCH for unknown species in the commercial whelk fishery, which has the scientific name STROMBIDAE and SPECIES_ITIS=72554. This maps to NESPP4=7750…FMRD has added a new code, NESPP4=7730 to our species tables, which maps to WHELK, MELONGENIDAE, NK and scientific name MELONGENIDAE. All existing records of 7750 north of 36 degrees latitude have been updated to 7730 to reflect the intention of the observers/monitors.”
Documentation
- Website and Documentation are now on the new server at http://nerswind/cams/
- Updated Discards documentation (Chapter 10)
Process Updates (News for Developers, does not affect outputs)
- Got frontend data load/staging/matching and landings running in a single script. Waiting on table transfer from DIS for automating all the way through discards. Planned to implement before the end of the year.
Notes
- Temporary Error: In CAMS_GARFO.CAMS_LANDINGS there was an error in the 2005-2021 data where some species such as lobster were not properly joined between DLR and VTR resulting in excessive dealer orphans. These orphans were then split between subtrips rather than being assigned to their correct subtrip. The total pounds for the trip was not affect but it would have affected the area, gear, and mesh assignments. It is unknown how many species or trips this affected. The error was likely in place from November 21 - 30. There is no expectation for this error to continue or reoccur.
MAPS 0.4.0
2022-11-09
SPECIAL NOTE: Please access the CAMS website via http://nersapsddev/cams/ and the documentation via http://nersapsddev/cams_documentation/ rather than the version that starts with the 155.206. IP address and only share these versions. Only these named versions can receive an automatic redirect when the website and full CAMS process moves to the new server. Once we move, we will put a link on the old 155.206. version to the new location but the user will have manually click it and even that will go away in 2023 when that old virtual server goes out of use.
Updates
- Fixed missing week in CAMS_LANDINGS for permit 000000
- Added offwatch haul info to CAMS_OBS_CATCH and DISCARD_ALL_YEARS
- Fixed bug that duplicated some VTR discards and vtr live pounds processing in CAMS_LANDINGS (did not affect livlb in the final landings or AA tables)
- Updated ITIS_GROUP1 to have more specific values since the focus is on skate DLR-VTR ID differences
- Fixed rounding issues for monkfish so that monkfish heads and liver livlb do not round up from zero to one
- Added date_run column to CAMS_LANDINGS
- Created a draft version of CAMS_SUBTRIP (misnomer) for consideration to later reduce CAMS_LANDINGS size
- Added DLR_RESEARCH column to CAMS_FISHERY_GROUP to indicate if the dealer reported the VTR starting with ‘8888’ to indicate a research trip outside the bounds of Magnuson-Stevens
Documentation
- Updated documentation to remove the “VTR” discard source description to match updates in previous release (no associated change in data in this release)
- Updated discard documentation details for improved clarity and to reflect current process
MAPS 0.3.0
2022-09-29
Updates
For Discards:
- Rebuilt discard tables and pushed to NOVA
- DISCARD_ALL_YEARS now contains EM trips in FY20, FY21 and FY22. Discard source for EM trips are EM for EM reviewed trips or Delta for unreviewed trips or trips that failed EM business rules.
- Adjusted code for observed trips with a single, unobserved, haul. These trips are now not counted toward the total observed trips within a strata. This has implications for transition rate calculations.
-
CAMS_OBS_CATCHusingLINK1in place ofOBS_LINK1(may have only been implemented as OBS_LINK1 on the GARFO side) -
CAMS_OBS_CATCHadded column flagging if at least one observed haul occurred on an observed tripLINK3_OBS -
CAMS_OBS_CATCH:ITIS_TSNnow used in place ofSPECIES_ITIS -
FISHDISP 090not used in discard rate calculations -
Observer Prgram Codes: Observed trips with program codes not appropriate for use in rate calculations for paricular spceis have been filtered for that purpose. - Fixed bug to reduced link1 drops (code change to match intent, no change to business rules or documentation)
- Changed vtrserno to subtrip in discard-link3 connection to drastically reduced dropped hauls (code change to match intent, no change to business rules or documentation)
- Added/renamed and adjusted types/precision/scale of support tables:
cfg_discard_runid,cfg_gearcode_strata,cfg_discard_mortality_stock,cfg_master_gear,cfg_observer_codes,cfg_statarea_stock
Other Updates:
- Added landings area assignment business rule: If the CAREA is outside the US EEZ but the reported AREA is inside the EEZ then use the reported area.
- Updated ITIS_GROUP0 in CFG_ITIS to have the coarser species groups (FIN, PEL, etc) previously reviewed as “spp_grp” which are used in the area and gear imputations. Until now ITIS_GROUP0 was mostly a placeholder for this and was primarily duplicate information of ITIS_GROUP1.
- Changed CFG_AREA to CFG_NEMAREA and added CFG_FED_AREA for use in herring QM and other area-specific needs
- Updated the directed loligo fishery group to match the regs (previously a placeholder). Now any trip landing >2500 lbs of loligo counts as a directed longfin trip which is used for Butterfish discard cap in QM
- Updated the EM field in CAMS_LANDINGS to include operational EM program ID beyond 2020. Categories did not change and no end user changes needed, just now inclusive of newer programs beyond EFPs.
- Fixed bug where the lat/lon seconds were getting dropped for dealer landing source subtrips but not VTR Orphans when creating lat_dd and lon_dd in CAMS_LANDINGS. Now lat/lon seconds are always used when creating decimal degrees.
- Fixed bug so VTR_LIVE and VTR_DISCARD carry through landings process regardless of status
- Updated herring fishery group to be more inclusive and also include state trips that land herring
Documentation
SPECIAL NOTE: Please access the CAMS website via http://nersapsddev/cams/ and the documentation via http://nersapsddev/cams_documentation/ rather than the version that starts with the 155. IP address and only share these versions. Only these named versions can receive an automatic redirect when the website and full CAMS process moves to the new server. Once we move we will put a link on the old 155. version to the new version but the user will have manually click it and even that will go away in 2023 when that server eventually goes out of use.
Documentation updated for:
CAMS_OBS_CATCH-
STG_OBDBS_YYYY(andALL_YEARS) CAMS_DISARD_ALL_YEARS- Discard chapter (currently, chapter 10)
- Added webpage for summarizing dropped trips and hauls in the observer data: http://155.206.47.98/cams/articles/summarize_obs_links.html
- Added examples and comparisons article to the website for observer length-frequency data :http://155.206.47.98/cams/articles/compare_length_age.html
Process Updates (News for Developers, does not affect outputs)
- Set up
configRunDefault.tomland adjusted the.onLoad()to create a copy asconfigRun.tomlin the users working directory if it does not exist. This config file can then be user-specific to adjust during testing or to run specific years. It can always be deleted and the default will be recreated the next time the MAPS package is called. The user-specific copy is ignored by git to prevent conflicts on github when testing or running different years locally. - Updated
zzz.R,setupFolders()andsetupConfig()to work with directory structures on either server - Set up folders on the wind server
- Added new convience paths
getOption("maps.landingsPath")andgetOption("maps.discardsPath")as subfolders of the output path - Updated the paths in landings scripts to pull from the internally defined paths in the
.onLoad()inzzz.R - Converted remaining landings code to use
roracle_login()to connect to the servers using ROracle and keyring for consistency across users - Set
setDTthreads(threads = 0)to use all available threads or as determined by OpenMP. It should be switched to 1 if running section in parallel to avoid nested parallel. - Converted a portion of discard scripts to functions and updated their output paths for full CAMS integration
MAPS 0.2.6
2022-07-20
Updates
- Fixed error in DLRID field that was introducing extra 0 unknowns
- Changed DLRID to character with padded 0. More appropriate data type and allows for better counting of the number of dealers for confidentiality (the rule of three). It also now matches the staged data in CAMS and the VTR_DLRID type. The change should NOT break any existing code since the length of the values is not changing and Oracle treats numeric characters as numbers in inequalities (e.g. DLRID > 0 and DLRID = 0 still work with ‘00000’).
- Built CAMS_FISHERY_GROUP: in review
- Updated priors for effort imputation to improve stability
- Added DOF-CML to the AMS stitching (so DOF-CML and other declarations can be part of a single trip when moving in and out of demarcation line)
MAPS 0.2.5
2022-06-02
Updates
- Updated source 08/11 handling to match new CFDERS data streams
- Updated area-adjacent port information
- Added CFLEN_DSM matching to trips (CAMSID)
- Added dockside-monitoring length (CFLEN_DSM) data to the CAMS_CFLENyyyyAA output tables
- Added STOCK_CT and MATCHED_STOCKS to CAMS_CFLENyyyyAA so data can be filtered to STOCK_CT = 1 for use in length-frequency and stock assessments
Bug Fixes
- Fixed if_else NA handling for gear imputation. VTR Orphans were get gears nullified and therefore imputed because of NA handling of dlr_source with if_else function. This also lead to extra subtrips being created. Switched to case_when and it is now handled as intended.
- EM designation of ‘MREM’ in CAMS_LANDINGS now only applied to groundfish trips
MAPS 0.2.4
2022-04-27
Updates
- The upstream dealer data was changed from the NEFSC copy on SOLE to FSO_ADMIN.CFDERS_ALL_YEARS on GARFO’s NERO.WORLD server. All years of CAMS data were rerun from this new upstream data source.
- Spring 2021 state data was added from ACCSP to CFDERS
- DLR_RPTID was expanded from 8 characters to 10 characters to handle upstream changes associated with SFCLAM landings in FSO_ADMIN.CFDERS_ALL_YEARS
- Added mixed hake apportionment based on red hake stock (REGION_HAKE) and mesh category (MESH_CAT)
- Improved NA handling associated with DLR_SOURCE for VTR Trip Orphans now prevents unnecessary dropping of certain VTR records from the area and gear imputation pools.
- The number of records required for level B gear imputation was lowered from 15 to 3 because we found the value of permit-specific information to outweigh the need for a larger sample size. The broader imputation level strata still require a minimum of 15 records in the strata to be used for imputation.
- Another permit-specific level was added as gear imputation level C (year, permit, main_spp_grp, dlr_source) to take advantage of more permit-specific information. There were cases when permits were being assigned gillnets when they only ever fish trawls because level B was too restrictive, such as cases when they had no VTR for that quarter of the year in that area. Going to fleet-level stratification then caused incompatible gears to be assigned to that permit. Although this cannot be completely prevented this new level C greatly reduces the chance. The other stratification levels each moved down one level (C -> D, D -> E, etc.)
- The number of records required for level B area imputation was lowered from 15 to 5 because we found the value of permit-specific information to outweigh the need for a larger sample size. The broader imputation level strata still require a minimum of 15 records in the strata to be used for imputation. The sample size is larger than for gear because of the bigger problem caused by imputing truly incompatible gear at the permit level.
- Added indexes to AA tables on CAMS_GARFO
MAPS 0.2.3
2022-04-06
Updates
- Converted if_else to case_when to better handle complex boolians with NA
- Converted missing spplivlb from NA/NULL to 0
- Output CAMS_LANDINGS with year index
- Output CAMS_LANDINGS_PARTITION with year partition for query speed comparison. Would have to be MUCH MUCH faster to justify the extra Oracle cost
MAPS 0.2.2
2022-03-31
Documentation
- Reorganized chapters for organization and to better reflect the changes in the processes
- New chapter 6: cleaning and joining landings data in preparation of imputation and apportionment
- Updated chapters 5-9 on matching, cleaning, imputation, apportionment, and AA tables
- Fixed a bug in the groundfish if_else function that led to NA and therefore dropping lat/lon and fzones from some records.
- Started a master data dictionary index that all tables can pull from to keep definitions consistent across tables and only have to update definitions in a single location. It also allows for a sortable, filterable index on the website
- Updated the key AA differences in Chapter 9. Still a work in progress as users give more feedback.
Updates
- Converted DLR_VTR and DLR_VTR_ZERO to one table: CAMS_LANDINGS
- Split Area imputation by source 05/07 for NOL/LOP trips
- Added Discards menu to the website with groundfish summary
- Added Monkfish declarations with AMS_PROG_CHRG > 0 to groundfish (GF) designation
- Removed international areas from the area imputation data pool for levels C-E.
- Added permit in addition to permit:month to the effort imputation model to better handle in-season imputation while VTRs are still coming in.
- Added rounding 0 lndlb, livlb, and value up to 1 after apportionment. Adjusted the higher subtrip down by an equal amount as long as it didn’t bring that below 1 to maintain the trip totals for each species.
- Calculated MAINSPP for CFDETT
- Increased N_SUBTRIP and SUBTRIP precision limits to accommodate 2 trips in 2013
- Increased aggregrate group precision limits in CFLEN and CFAGE
- Added DATE_RUN to STG_GDP_DEFLATOR so can auto update once every 30 days
- Added overlap of a month on each side of the run year to handle dealer date from the year after the land date.
- Added PERMIT_STATE_FED to CAMS_LANDINGS as an easy way to separate and group state and federal permits since it is such a common part of analyses and requests.
- Added CAMS_LANDINGS_METADATA table to track state data and summary information
Technical Updates
- Added call to apsdFuns package for new roracle_login and odbc_login using keyring for easy, secure connection setup
- Added partition check to DLR_VTR/CAMS_LANDINGS oracle push to make the function more robust
- Added all years to the STG_TRIP_ATTR view
- Added skip for pushing empty rec data to Oracle in loop
- Added logfile for landings process
- Suppressed brms iteration output
- Separated permit zero dealer handling into own function
- Setup landings to loop through years
- Set so only create AA tables for years >= 2019 to save time and DB space in other year runs
MAPS 0.2.0
2022-02-14
Happy Valentines Day :)
Major Changes
- For federal permits (NOL and LOP), now use VTR -> IMPUTE -> DLR -> PORT
- GEAR impute missing permit 000000 data by levels C, D, E (GEAR_SOURCE = ‘IMPUTE_STATE’)
- Expanded GEAR_SOURCE to VARCHAR2(12 BYTE) to accomodate new source name
- If DLR Orphan and source 05 then prioritize dealer then VTR then impute (e.g. 2020 trip 214887_20200718140000_5326767). DLR species orphans are sometimes not recorded on VTR and use different gear that are not refleted on the VTR they submitted for (match) federal species. Therefore those non-federal species or species in state waters that are not on the VTR have gear better reflected by the dealer then the trip VTR. For example, with CAMSID = 214887_20200718140000_5326767 they had lobster and other federal permits active on the day of the trip but only subimtted a VTR for the lobster landed but the dealer also had thousands of pounds of Menhaden and Herring that the dealer listed as coming from purse seine with source 05. Previously these would have used the VTR gear of lobster traps (NEGEAR 200). Now it gets a separate subtrip for the source 05 dealer orphans with dealer gear 120 (purse seine) for the menhaden/herring landings. This was an extreme edge case but now makes more sense.
- Fixed a minor error in handling subtrip apportionment for dealer species orphans related to the above change. This fixed a small bug so that 2019 dealer pounds in CAMS AA tables are within ~200 of CFDERS rather than within ~2500 lbs.
Minor Changes
- Expanded DF and DA from two to four decimal places. This fixed the issue of adjusting values when the DF was small but the number of transactions was relatively large.
- Now round up DF < 0.00005 to 0.0001 in order to prevent assigning zero effort.
- Changed link to an index because the concatenated string was getting to long and just duplicating information in other fields.
- Summarized CFDETS to all columns except lndlb, livlb, and value to reduce split weighouts
- Added null/unknown comparison bewteen CAMS and AA tables: http://155.206.47.98/cams/articles/articles/check_nulls.html
MAPS 0.1.7
2022-02-06
Major Changes
- Using port adjacent statistical areas for state permits (permit 000000)
- Added NOP to LOBSTER_CLASS to denote trips with permits not assigned any federally species during the trip - Assumed NOP acting as permit 000000 during the trip and therefore get area adjacent to the port and DLR NEGEAR, unless they submitted a VTR (VTR area and gear always prioritized). This represents a large percentage of trips with federal dealer permits.
- Changed recreational designation to exclude landings sold to federal dealers. Now CFDERS landings will all be in DLR_VTR and CFDETT/S and VTR_REC table only includes VTR landings on recreational trips that do not match to DLRs. Therefore CFDETT/S total pounds where LANDING_SOURCE = ‘DLR’ should equal the total pounds in CFDERS (+/- 1 MT).
Additional Changes
- Added GDP Deflator code to create STG_GDP_DEFLATOR. Not yet used by any tables, just staged for future use.
- Created CFG_PORT_AREA for use in assigning area to LOO and State trips based on port. This is a conversion of a preivous APSD table that had port names rather than numbers. The goal was to bring it into CAMS where it can have ports added and not have to remake it on the fly from an unmaintained table.
- Added recreational imputation and apportionment (separate from commercial)
- Added rec output table (VTR_REC)
- Added MAINSPP placeholder to CFDETT
- Moved LOBSTER_CLASS from CFDETS to CFDETT because it is a trip-level attribute
MAPS 0.1.5
2022-01-26
- Fixed CFDERS duplication issues
- Fixed port_grp for ports not in port_grp.csv to prevent join errors
- Added STATE to AREA imputation level E but now leaves a handful of AREA as 000 from florida where no VTR were ever submitted from and also no DLR area reported.
- Fixed NA handling in Lobster_Class designation
- Source 08 (clam logbook) NEGEAR and AREA from CFDERS
- Added tallyno and sampno to CFLEN
MAPS 0.1.4
2022-01-21
Changes
- CFG_ITIS: fixed two itis_tsn mapping to names and groups that could cause duplicated depending on use
- Additional apportionment-imputation checks for changes in dealer pounds
- Cleaned lon to be between 50 and 100 (West is implied)
- Cleaned lat to be between 20 and 50
- Changed latdeg, latmin, londeg, and lonmin to coalesce calculated and reported lat/lon depending on groundfish trip status
- Converted LAT/LON NULLs to 00
- Added LAT and LON
- Replaced “CLOSEST” area and gear imputation with level B (quarter, main_spp_grp, permit) to add uncertainty and avoids potential issues with switch gears and mis-reported gears.
- Updated nespp3_spp_grp.csv file with recommendations by CAMS team to better handle gear imputation for particular species (menhaden, scallops, lobster, etc.)
- Added species-gears summary to website
- Removed NEGEAR 999 out of imputation set
- Added STATE, SPRATIO, and SAMPAREA to CFDETS
- Added PORT2 to CFDETT
- Added MESH_SOURCE to CFDETS and CFDETT
- Renames GEAR_QTY and GEAR_SIZE without underscore in AA tables
- Added imputation-summary and species-gear summaries to the website under Articles
MAPS 0.1.3
2022-01-10
Changes
- Mesh categories updated to remove medium size class (MM). Now gillnets >8 = XL, 4.00-8.00 = LM, < 4.00 = SM. Trawl >= 4.00 = LM, < 4.00 = SM.
- Matching processing updated to avoid dropping pounds when handling dealer combined trips
- Internal NA handling in boolian updated to avoid a minor duplication issue with apportionment
MAPS 0.1.2
2021-12-16
Changes
- Reran staging and matching (JMLanning)
- Added NEGEAR 115 to the XL mesh category
- Changed DOE in CFDETT/S to be the max associated with the link to fix the final 36 records in 2019 causing cartesians when rejoining the two
- Fixed bug where days absent were being incorrectly calculated for subtrips
- Imputation by stratification level for AREA, NEGEAR, and MESH_CAT now must have at least 15 records in the stratification group to be considered or progresses to the next stratification level (larger groupings). This follows previous AA methods to avoid spurious imputations.
- Updated Imputation Documentation to reflect recent updates since version 0.0.1-9000