Title
#core
s

seph

08/16/2021, 1:29 AM
There’s something weird with the app bundle:
miniseph:o seph$ /usr/sbin/spctl -a -vvv  --ignore-cache opt/osquery/lib/osquery.app
opt/osquery/lib/osquery.app: code has no resources but signature indicates they must be present
1:32 AM
This is status/error: -67056:
code has no resources but signature indicates they must be present. Most probably structural damage, which should be referred to the developer
(says https://eclecticlight.co/2019/06/28/a-primer-on-code-signing-errors/)
1:35 AM
I’m guessing this is because we sign the binary, but not the .app
1:53 AM
There’s so much tied together it’s hard for me to quickly resign and test this
2:02 AM
Okay. I went through resigning. A pretty normal process, I used the same codesign commands that we do with the release file. but against the
.app
. The notarized it. Now it shows as signed and entitled. I’m less sure how to fix the build to handle this, I think we’re dumping raw stuff into package-data, which is then signed, and then packaged. And the packaging step converts it to a .app. But I think that ordering is probably wrong. Instead, we should be shoving the .app into the package data. Which can then be signed, and packaged into to the tar.gz and pkg. (I’d probably say the package-data should basically be identical to the tar.gz)
2:05 AM
I even kinda understand stuff. With status quo, I can run osqueryd from contents, but I’m skeptical it has the entitlements, and it generally looks unsigned. But, I can move it outside the .app bundle, and then it starts looking pretty normal.
2:05 AM
With my signing the .app. Things display as codesigned and notarized. But if I copy the binary out of the app, it complains about being invalid
2:09 AM
Which, ultimately, probably means we need to view the app bundle as somewhat more real than we thought. I think we can distribute a bare, signed, osqueryd. But it may not be as useful.
2:22 AM
If anyone wants to play, here’s a zip. This was built on CI, but signed on my local machine.
theopolis

theopolis

08/16/2021, 2:43 AM
I am not sure I am following exactly, but it's late and my brain is most likely fried. Are we OK if we sign the binary, then the app bundle? I would expect that allows the binary to be moved out of the bundle and used as a normal signed binary.
2:46 AM
But I see what you are saying about https://github.com/osquery/osquery-codesign/blob/master/.github/workflows/release-generator.yml#L417 needing to sign the app. Let me think through this and see if there's an easy path.
2:50 AM
I don't see a downside to the package_data including the app bundle layout. That means less logic in osquery-packaging
a

alessandrogario

08/16/2021, 11:42 AM
Additional note to why package_data is done like that: since not all package formats require the same files, we used a neutral archive that contrains a 'control' folder with all the files required for all different formats
11:43 AM
One of the major issues we had before is that we couldn't generate an .MSI file from a the ZIP release because the WiX data was missing. The opposite was also true, we couldn't start from the .MSI to generate Choco because we were missing the nupkg configuration files
11:45 AM
I think it is fine to include .app as an additional format if there is no other way, but I think it makes sense to keep package_data neutral and see if we can fix this the correct way (and keep the standard procedure to sign while the package is being made)
s

sharvil

08/16/2021, 11:46 AM
Taking a look...
s

seph

08/16/2021, 1:28 PM
Are we OK if we sign the binary, then the app bundle?
From what I can tell, signing the app bundle overwrites the sig on the plain macho binary.
s

sharvil

08/16/2021, 1:55 PM
I think the
spctl
output is a red-herring of sorts (while not ideal)... since our app bundle is a minimal set of things required for it to work I took the
osquery-a08056b68ef64b23d163fafdfe6f0e1681c80a5c.pkg
from the test s3 bucket, installed it on a fresh macOS 11 VM, and things are working fine. It is picking up the entitlements and also the provisioning profile correctly, and I can see
es_process_events
flowing in
1:55 PM
I am still digging a bit into this...
s

seph

08/16/2021, 1:58 PM
I could run the
osqueryd
from the the bundle, but all the signature verification tools I have say it’s a bad sig. Which makes me think we’re asking for trouble,
s

sharvil

08/16/2021, 2:00 PM
any other tool apart from spctl?
s

seph

08/16/2021, 2:01 PM
s

sharvil

08/16/2021, 2:01 PM
root@osquerytests-Mac # codesign -vvv --strict --deep --display /opt/osquery/lib/osquery.app
Executable=/opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
Identifier=osqueryd
Format=app bundle with Mach-O thin (x86_64)
CodeDirectory v=20500 size=311860 flags=0x10000(runtime) hashes=9737+5 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=870a7d77eabc2d12c410060aed030bbe940a81fc
CandidateCDHashFull sha256=870a7d77eabc2d12c410060aed030bbe940a81fc94241ab9824f21f9a96f7348
Hash choices=sha256
CMSDigest=870a7d77eabc2d12c410060aed030bbe940a81fc94241ab9824f21f9a96f7348
CMSDigestType=2
CDHash=870a7d77eabc2d12c410060aed030bbe940a81fc
Signature size=9021
Authority=Developer ID Application: OSQUERY A Series of LF Projects, LLC (3522FA9PXF)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Aug 15, 2021 at 5:52:53 PM
Info.plist=not bound
TeamIdentifier=3522FA9PXF
Runtime Version=11.1.0
Sealed Resources=none
Internal requirements count=1 size=168
root@osquerytests-Mac _CodeSignature #
s

seph

08/16/2021, 2:05 PM
dover:osq5 seph$ codesign -vvv --strict --deep opt/osquery/lib/osquery.app
opt/osquery/lib/osquery.app: code has no resources but signature indicates they must be present
s

sharvil

08/16/2021, 2:06 PM
I see there is a new
_CodeSignature/CodeResources
file in the signed app bundle
2:06 PM
root@osquerytests-Mac Desktop # find ~/Desktop/osquery.app
/Users/osquerytest/Desktop/osquery.app
/Users/osquerytest/Desktop/osquery.app/Contents
/Users/osquerytest/Desktop/osquery.app/Contents/_CodeSignature
/Users/osquerytest/Desktop/osquery.app/Contents/_CodeSignature/CodeResources
/Users/osquerytest/Desktop/osquery.app/Contents/MacOS
/Users/osquerytest/Desktop/osquery.app/Contents/MacOS/osqueryd
/Users/osquerytest/Desktop/osquery.app/Contents/Resources
/Users/osquerytest/Desktop/osquery.app/Contents/Resources/osqueryctl
/Users/osquerytest/Desktop/osquery.app/Contents/embedded.provisionprofile
/Users/osquerytest/Desktop/osquery.app/Contents/Info.plist
/Users/osquerytest/Desktop/osquery.app/Contents/PkgInfo
root@osquerytests-Mac Desktop # find /opt/osquery/lib/osquery.app 
/opt/osquery/lib/osquery.app
/opt/osquery/lib/osquery.app/Contents
/opt/osquery/lib/osquery.app/Contents/MacOS
/opt/osquery/lib/osquery.app/Contents/MacOS/osqueryd
/opt/osquery/lib/osquery.app/Contents/Resources
/opt/osquery/lib/osquery.app/Contents/Resources/osqueryctl
/opt/osquery/lib/osquery.app/Contents/embedded.provisionprofile
/opt/osquery/lib/osquery.app/Contents/Info.plist
/opt/osquery/lib/osquery.app/Contents/PkgInfo
2:07 PM
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "<http://www.apple.com/DTDs/PropertyList-1.0.dtd>">
<plist version="1.0">
<dict>
	<key>files</key>
	<dict>
		<key>Resources/osqueryctl</key>
		<data>
		cfmKsVPNeZNg+/+Phdj1bSeUOE4=
		</data>
	</dict>
	<key>files2</key>
	<dict>
		<key>Resources/osqueryctl</key>
		<dict>
			<key>hash2</key>
			<data>
			XK+14UzR5NpyldAfI3YrGoH2v0L1pzZLt/Hb6yBA5ko=
			</data>
		</dict>
		<key>embedded.provisionprofile</key>
		<dict>
			<key>hash2</key>
			<data>
			X1tITOEGCKY4oqba+4SFfq3vhZW3rGSe8gjNcIk9t1M=
			</data>
		</dict>
	</dict>
	<key>rules</key>
	<dict>
		<key>^Resources/</key>
		<true/>
		<key>^Resources/.*\.lproj/</key>
		<dict>
			<key>optional</key>
			<true/>
			<key>weight</key>
			<real>1000</real>
		</dict>
		<key>^Resources/.*\.lproj/locversion.plist$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>1100</real>
		</dict>
		<key>^Resources/Base\.lproj/</key>
		<dict>
			<key>weight</key>
			<real>1010</real>
		</dict>
		<key>^version.plist$</key>
		<true/>
	</dict>
	<key>rules2</key>
	<dict>
		<key>.*\.dSYM($|/)</key>
		<dict>
			<key>weight</key>
			<real>11</real>
		</dict>
		<key>^(.*/)?\.DS_Store$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>2000</real>
		</dict>
		<key>^(Frameworks|SharedFrameworks|PlugIns|Plug-ins|XPCServices|Helpers|MacOS|Library/(Automator|Spotlight|LoginItems))/</key>
		<dict>
			<key>nested</key>
			<true/>
			<key>weight</key>
			<real>10</real>
		</dict>
		<key>^.*</key>
		<true/>
		<key>^Info\.plist$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^PkgInfo$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^Resources/</key>
		<dict>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^Resources/.*\.lproj/</key>
		<dict>
			<key>optional</key>
			<true/>
			<key>weight</key>
			<real>1000</real>
		</dict>
		<key>^Resources/.*\.lproj/locversion.plist$</key>
		<dict>
			<key>omit</key>
			<true/>
			<key>weight</key>
			<real>1100</real>
		</dict>
		<key>^Resources/Base\.lproj/</key>
		<dict>
			<key>weight</key>
			<real>1010</real>
		</dict>
		<key>^[^/]+$</key>
		<dict>
			<key>nested</key>
			<true/>
			<key>weight</key>
			<real>10</real>
		</dict>
		<key>^embedded\.provisionprofile$</key>
		<dict>
			<key>weight</key>
			<real>20</real>
		</dict>
		<key>^version\.plist$</key>
		<dict>
			<key>weight</key>
			<real>20</real>
		</dict>
	</dict>
</dict>
</plist>
Which is a plist of all files and "hashes"
2:14 PM
yep, you are right! we should sign the
.app
bundle too
s

seph

08/16/2021, 2:14 PM
AFAICT there’s no point to signging the macho binary prior.
s

sharvil

08/16/2021, 2:14 PM
okay, I am gonna put a PR to try things out
2:15 PM
might result in some failed workflows and notifications while i try things out
2:15 PM
yes, I think so too
s

seph

08/16/2021, 2:15 PM
I can review PRs intermittently.
2:16 PM
But I think it would be clean to have the .app generated in the osquery package data phase
a

alessandrogario

08/16/2021, 2:45 PM
reading again, i can see your point
2:45 PM
makes sense to do it that way
2:46 PM
i was initially assuming the .app to be a package itself but it's totally wrong as it's in fact a binary
s

seph

08/16/2021, 4:06 PM
Yeah. The .app is a special directory structure around some binaries. It’s sorta like a package, in you might not need any more.
s

sharvil

08/16/2021, 5:35 PM
5:35 PM
working on updating packaging repo next
6:44 PM
would like your code reviews, whenever you get a moment @seph @theopolis